Giter Club home page Giter Club logo

pihole_adlist_tool's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pihole_adlist_tool's Issues

Shellcheck is complaining a lot about pihole_adlist_tool

Missing shellcheck compatibility is no fun because there are some severe and critical errors - and this breaks my pipeline because I copied your script in my bin directory which is shellcheck'ed on every commit:

grafik

("passed" now with my changes from PR #37 )

First round:

shellcheck pihole_adlist_tool
pihole_adlist_tool: <stdout>: commitBuffer: invalid argument (invalid character)

In pihole_adlist_tool line 87:
	local remote_version="$(
              ^------------^ SC2155: Declare and assign separately to avoid masking return values.


In pihole_adlist_tool line 125:
    local DATE=$(date -d @$1 +%d.%m.%Y%t%k:%M:%S)
          ^--^ SC2155: Declare and assign separately to avoid masking return values.
                          ^-- SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    local DATE=$(date -d @"$1" +%d.%m.%Y%t%k:%M:%S)


In pihole_adlist_tool line 134:
    TIMESTAMP_FIRST_QUERY=$(sqlite $PIHOLE_FTL "SELECT MIN(timestamp) FROM queries;")
                                   ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_FIRST_QUERY=$(sqlite "$PIHOLE_FTL" "SELECT MIN(timestamp) FROM queries;")


In pihole_adlist_tool line 135:
    TIMESTAMP_LAST_QUERY=$(sqlite $PIHOLE_FTL "SELECT MAX(timestamp) FROM queries;")
                                  ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_LAST_QUERY=$(sqlite "$PIHOLE_FTL" "SELECT MAX(timestamp) FROM queries;")


In pihole_adlist_tool line 141:
            TIMESTAMP_REQUESTED=$TIMESTAMP_REQUESTED-86400*${DAYS_REQUESTED}
                                ^-- SC2125: Brace expansions and globs are literal in assignments. Quote it or use an array.


In pihole_adlist_tool line 143:
    TIMESTAMP_FIRST_ANALYZED=$(sqlite $PIHOLE_FTL "SELECT min(timestamp) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")
                                      ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_FIRST_ANALYZED=$(sqlite "$PIHOLE_FTL" "SELECT min(timestamp) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")


In pihole_adlist_tool line 149:
    DATE_FIRST_QUERY="$(timestamp2date $TIMESTAMP_FIRST_QUERY)"
                                       ^--------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_FIRST_QUERY="$(timestamp2date "$TIMESTAMP_FIRST_QUERY")"


In pihole_adlist_tool line 150:
    DATE_LAST_QUERY="$(timestamp2date $TIMESTAMP_LAST_QUERY)"
                                      ^-------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_LAST_QUERY="$(timestamp2date "$TIMESTAMP_LAST_QUERY")"


In pihole_adlist_tool line 151:
    DATE_REQUESTED="$(timestamp2date $TIMESTAMP_REQUESTED)"
                                     ^------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_REQUESTED="$(timestamp2date "$TIMESTAMP_REQUESTED")"


In pihole_adlist_tool line 152:
    DATE_FIRST_ANALYZED="$(timestamp2date $TIMESTAMP_FIRST_ANALYZED)"
                                          ^-----------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_FIRST_ANALYZED="$(timestamp2date "$TIMESTAMP_FIRST_ANALYZED")"


In pihole_adlist_tool line 171:
echo "  [For more information:
  https://www.shellcheck.net/wiki/SC2068 -- Double quote array expansions to ...
  https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ...
  https://www.shellcheck.net/wiki/SC2027 -- The surrounding quotes actually u...

After removing "invalid character"(s):

In pihole_adlist_tool line 87:
	local remote_version="$(
              ^------------^ SC2155: Declare and assign separately to avoid masking return values.


In pihole_adlist_tool line 125:
    local DATE=$(date -d @$1 +%d.%m.%Y%t%k:%M:%S)
          ^--^ SC2155: Declare and assign separately to avoid masking return values.
                          ^-- SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    local DATE=$(date -d @"$1" +%d.%m.%Y%t%k:%M:%S)


In pihole_adlist_tool line 134:
    TIMESTAMP_FIRST_QUERY=$(sqlite $PIHOLE_FTL "SELECT MIN(timestamp) FROM queries;")
                                   ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_FIRST_QUERY=$(sqlite "$PIHOLE_FTL" "SELECT MIN(timestamp) FROM queries;")


In pihole_adlist_tool line 135:
    TIMESTAMP_LAST_QUERY=$(sqlite $PIHOLE_FTL "SELECT MAX(timestamp) FROM queries;")
                                  ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_LAST_QUERY=$(sqlite "$PIHOLE_FTL" "SELECT MAX(timestamp) FROM queries;")


In pihole_adlist_tool line 141:
            TIMESTAMP_REQUESTED=$TIMESTAMP_REQUESTED-86400*${DAYS_REQUESTED}
                                ^-- SC2125: Brace expansions and globs are literal in assignments. Quote it or use an array.


In pihole_adlist_tool line 143:
    TIMESTAMP_FIRST_ANALYZED=$(sqlite $PIHOLE_FTL "SELECT min(timestamp) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")
                                      ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    TIMESTAMP_FIRST_ANALYZED=$(sqlite "$PIHOLE_FTL" "SELECT min(timestamp) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")


In pihole_adlist_tool line 149:
    DATE_FIRST_QUERY="$(timestamp2date $TIMESTAMP_FIRST_QUERY)"
                                       ^--------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_FIRST_QUERY="$(timestamp2date "$TIMESTAMP_FIRST_QUERY")"


In pihole_adlist_tool line 150:
    DATE_LAST_QUERY="$(timestamp2date $TIMESTAMP_LAST_QUERY)"
                                      ^-------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_LAST_QUERY="$(timestamp2date "$TIMESTAMP_LAST_QUERY")"


In pihole_adlist_tool line 151:
    DATE_REQUESTED="$(timestamp2date $TIMESTAMP_REQUESTED)"
                                     ^------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_REQUESTED="$(timestamp2date "$TIMESTAMP_REQUESTED")"


In pihole_adlist_tool line 152:
    DATE_FIRST_ANALYZED="$(timestamp2date $TIMESTAMP_FIRST_ANALYZED)"
                                          ^-----------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    DATE_FIRST_ANALYZED="$(timestamp2date "$TIMESTAMP_FIRST_ANALYZED")"


In pihole_adlist_tool line 171:
echo "  [i]  Of the ${bold}"$NUM_DOMAINS_BLOCKED_CURRENT"${normal} domains that would have been blocked by the adlist configuration you chose to analyze,"
                            ^--------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                            ^--------------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  Of the ${bold}""$NUM_DOMAINS_BLOCKED_CURRENT""${normal} domains that would have been blocked by the adlist configuration you chose to analyze,"


In pihole_adlist_tool line 172:
echo "       ${bold}"$NUM_DOMAINS_BLOCKED_FUTURE"${normal} domains are covered by your current adlist configuration."
                     ^-------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                     ^-------------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "       ${bold}""$NUM_DOMAINS_BLOCKED_FUTURE""${normal} domains are covered by your current adlist configuration."


In pihole_adlist_tool line 184:
        docker exec $CONTAINER_ID rm -f $TEMP_DB
                    ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        docker exec "$CONTAINER_ID" rm -f $TEMP_DB


In pihole_adlist_tool line 205:
        docker exec -i $CONTAINER_ID pihole-FTL sqlite3 "$@"
                       ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        docker exec -i "$CONTAINER_ID" pihole-FTL sqlite3 "$@"


In pihole_adlist_tool line 260:
        PIHOLE_DNSMASQ_VERSION=$(docker exec $CONTAINER_ID pihole-FTL -vv |awk '/dnsmasq/{getline; print substr($2,9)}')
                                             ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        PIHOLE_DNSMASQ_VERSION=$(docker exec "$CONTAINER_ID" pihole-FTL -vv |awk '/dnsmasq/{getline; print substr($2,9)}')


In pihole_adlist_tool line 305:
        SQLITE_VERSION=$(docker exec $CONTAINER_ID pihole-FTL sqlite3 --version|awk '{print $1}')
                                     ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        SQLITE_VERSION=$(docker exec "$CONTAINER_ID" pihole-FTL sqlite3 --version|awk '{print $1}')


In pihole_adlist_tool line 360:
if [ -n "$remote_version" -a "$remote_version" != "$PIHOLE_ADLIST_TOOL_VERSION" ]; then
                          ^-- SC2166: Prefer [ p ] && [ q ] as [ p -a q ] is not well defined.


In pihole_adlist_tool line 386:
FTL_ID=$(sqlite $PIHOLE_FTL "SELECT MIN(id) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")
                ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
FTL_ID=$(sqlite "$PIHOLE_FTL" "SELECT MIN(id) FROM queries WHERE timestamp>=$TIMESTAMP_REQUESTED;")


In pihole_adlist_tool line 394:
        echo "  [i]  ${bold}Warning:${normal} You requested to analyze the last "${DAYS_REQUESTED}" day(s) (starting from "$DATE_REQUESTED"),"
                                                                                 ^---------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                 ^---------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                           ^-------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                                           ^-------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "  [i]  ${bold}Warning:${normal} You requested to analyze the last ""${DAYS_REQUESTED}"" day(s) (starting from ""$DATE_REQUESTED""),"


In pihole_adlist_tool line 395:
        echo "       but last query is from "$DATE_LAST_QUERY""
                                             ^--------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                             ^--------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "       but last query is from ""$DATE_LAST_QUERY"""


In pihole_adlist_tool line 404:
        echo -e "  [i]  ${bold}Warning:${normal} You requested to analyze the last "${DAYS_REQUESTED}" days (starting from "$DATE_REQUESTED"),"
                                                                                    ^---------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                    ^---------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                            ^-------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                                            ^-------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo -e "  [i]  ${bold}Warning:${normal} You requested to analyze the last ""${DAYS_REQUESTED}"" days (starting from ""$DATE_REQUESTED""),"


In pihole_adlist_tool line 405:
        echo -e "       but oldest query is from "$DATE_FIRST_QUERY". Using this instead\n"
                                                  ^---------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                  ^---------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo -e "       but oldest query is from ""$DATE_FIRST_QUERY"". Using this instead\n"


In pihole_adlist_tool line 411:
adlist_conf_old_enabled=(`sqlite $GRAVITY "select id from adlist where enabled=1;"`)
                         ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                         ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
adlist_conf_old_enabled=($(sqlite $GRAVITY "select id from adlist where enabled=1;"))


In pihole_adlist_tool line 426:
  read -p "  Please select: " ENABLE_ALL_ADLISTS_FOR_ANALYSIS
  ^--^ SC2162: read without -r will mangle backslashes.


In pihole_adlist_tool line 441:
    adlist_enabled_in_gravity=(`sqlite $GRAVITY "select distinct adlist_id from gravity;"`)
                               ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                               ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
    adlist_enabled_in_gravity=($(sqlite $GRAVITY "select distinct adlist_id from gravity;"))


In pihole_adlist_tool line 442:
    if [ -n "$(echo ${adlist_conf_old_enabled[@]} ${adlist_enabled_in_gravity[@]} | tr ' ' '\n' |sort |uniq -u)" ]; then
                    ^---------------------------^ SC2068: Double quote array expansions to avoid re-splitting elements.
                                                  ^-----------------------------^ SC2068: Double quote array expansions to avoid re-splitting elements.


In pihole_adlist_tool line 446:
        echo "       You have ${bold}"${#adlist_conf_old_enabled[@]}"${normal} adlists enabled, but data from ${bold}"${#adlist_enabled_in_gravity[@]}"${normal} (patially different) adlists in your gravity database."
                                      ^----------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                                      ^-- SC2027: The surrounding quotes actually unquote this. Remove or escape them.


In pihole_adlist_tool line 463:
            read -p "  Please select: " RUN_GRAVITY_NOW
            ^--^ SC2162: read without -r will mangle backslashes.


In pihole_adlist_tool line 474:
                        docker exec $CONTAINER_ID pihole -g
                                    ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
                        docker exec "$CONTAINER_ID" pihole -g


In pihole_adlist_tool line 513:
           docker exec $CONTAINER_ID pihole -g
                       ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
           docker exec "$CONTAINER_ID" pihole -g


In pihole_adlist_tool line 637:
read NUM_DOMAINS_BLOCKED HITS_TOTAL <<<$(sqlite -separator " " $PIHOLE_FTL "SELECT COUNT(DISTINCT domain),count(domain) FROM queries WHERE id>=${FTL_ID} AND status in (1,9);")
^--^ SC2162: read without -r will mangle backslashes.
                                       ^-- SC2046: Quote this to prevent word splitting.
                                                               ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
read NUM_DOMAINS_BLOCKED HITS_TOTAL <<<$(sqlite -separator " " "$PIHOLE_FTL" "SELECT COUNT(DISTINCT domain),count(domain) FROM queries WHERE id>=${FTL_ID} AND status in (1,9);")


In pihole_adlist_tool line 650:
echo "  [i]  You have ${bold}"$NUM_ADLISTS" adlists${normal} configured ("$NUM_ADLISTS_ENABLED" enabled)"
                              ^----------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                              ^----------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                          ^------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                          ^------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  You have ${bold}""$NUM_ADLISTS"" adlists${normal} configured (""$NUM_ADLISTS_ENABLED"" enabled)"


In pihole_adlist_tool line 651:
echo "  [i]  Your gravity.db contains ${bold}"$NUM_GRAVITY_UNIQUE_DOMAINS" unique domains${normal}"
                                              ^-------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                              ^-------------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  Your gravity.db contains ${bold}""$NUM_GRAVITY_UNIQUE_DOMAINS"" unique domains${normal}"


In pihole_adlist_tool line 652:
echo "  [i]  Since "$DATE_FIRST_ANALYZED" ${bold}"$NUM_DOMAINS_BLOCKED" different domains${normal} from your adlists have been blocked ${bold}"$HITS_TOTAL" times${normal} in total"
                    ^------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                    ^------------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                  ^------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                  ^------------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                                               ^---------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                                                               ^---------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  Since ""$DATE_FIRST_ANALYZED"" ${bold}""$NUM_DOMAINS_BLOCKED"" different domains${normal} from your adlists have been blocked ${bold}""$HITS_TOTAL"" times${normal} in total"


In pihole_adlist_tool line 654:
echo "  [i]  Using you current adlist configuration ${bold}"$NUM_DOMAINS_BLOCKED_CURRENT" domains${normal} would have been blocked ${bold}"$HITS_TOTAL_CURRENT" times${normal}"
                                                            ^--------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                            ^--------------------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                                           ^-----------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                                                           ^-----------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  Using you current adlist configuration ${bold}""$NUM_DOMAINS_BLOCKED_CURRENT"" domains${normal} would have been blocked ${bold}""$HITS_TOTAL_CURRENT"" times${normal}"


In pihole_adlist_tool line 706:
        echo "       Those would have been the ${bold}"$TOP" top blocked adlist domains${normal} since "$DATE_FIRST_ANALYZED""
                                                       ^--^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                       ^--^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                        ^------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                                        ^------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "       Those would have been the ${bold}""$TOP"" top blocked adlist domains${normal} since ""$DATE_FIRST_ANALYZED"""


In pihole_adlist_tool line 709:
        sqlite -column -header $TEMP_DB "SELECT domain, hits FROM blocked_domains LIMIT "${TOP}";"
                                                                                         ^----^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                         ^----^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        sqlite -column -header $TEMP_DB "SELECT domain, hits FROM blocked_domains LIMIT ""${TOP}"";"


In pihole_adlist_tool line 735:
echo "  [i]  In total your adlists contain ${bold}"$NUM_TOTAL_UNIQUE_DOMAINS" visited (covered) unique domains${normal} - meaning those domains are contained only in a single adlist. "
                                                   ^-----------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                   ^-----------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
echo "  [i]  In total your adlists contain ${bold}""$NUM_TOTAL_UNIQUE_DOMAINS"" visited (covered) unique domains${normal} - meaning those domains are contained only in a single adlist. "


In pihole_adlist_tool line 756:
          read -p "  Please select: " FURTHER_ACTION
          ^--^ SC2162: read without -r will mangle backslashes.


In pihole_adlist_tool line 773:
          read -p "  Please select: " FURTHER_ACTION
          ^--^ SC2162: read without -r will mangle backslashes.


In pihole_adlist_tool line 792:
    adlist_conf_unique_enabled=(`sqlite $TEMP_DB "select id from adlist where unique_domains_covered IS NOT NULL;"`)
                                ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                                ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
    adlist_conf_unique_enabled=($(sqlite $TEMP_DB "select id from adlist where unique_domains_covered IS NOT NULL;"))


In pihole_adlist_tool line 807:
            docker exec $CONTAINER_ID pihole restartdns reload-lists
                        ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
            docker exec "$CONTAINER_ID" pihole restartdns reload-lists


In pihole_adlist_tool line 836:
    adlist_conf_minimal_enabled=(`sqlite $TEMP_DB "select id from adlist where unique_domains_covered IS NOT NULL;"`)
                                 ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                                 ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
    adlist_conf_minimal_enabled=($(sqlite $TEMP_DB "select id from adlist where unique_domains_covered IS NOT NULL;"))


In pihole_adlist_tool line 844:
    LEFT_DOMAINS=(`sqlite $TEMP_DB "SELECT COUNT (domain) from gravity_dup;"`)
                  ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                  ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
    LEFT_DOMAINS=($(sqlite $TEMP_DB "SELECT COUNT (domain) from gravity_dup;"))


In pihole_adlist_tool line 846:
    while [[ $LEFT_DOMAINS != [0] ]]; do
             ^-----------^ SC2128: Expanding an array without an index only gives the first element.


In pihole_adlist_tool line 847:
        current_id=(`sqlite $TEMP_DB "Select adlist_id from gravity_dup group by adlist_id order by count (domain) desc, adlist_id asc limit 1;"`);
                    ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                    ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
        current_id=($(sqlite $TEMP_DB "Select adlist_id from gravity_dup group by adlist_id order by count (domain) desc, adlist_id asc limit 1;"));


In pihole_adlist_tool line 849:
        adlist_conf_minimal_enabled[${#adlist_conf_minimal_enabled[@]}]="$current_id"
                                                                         ^---------^ SC2128: Expanding an array without an index only gives the first element.


In pihole_adlist_tool line 850:
        sqlite $TEMP_DB "DELETE FROM gravity_dup WHERE domain IN (SELECT domain from gravity_dup where adlist_id=$current_id);"
                                                                                                                 ^---------^ SC2128: Expanding an array without an index only gives the first element.


In pihole_adlist_tool line 851:
        LEFT_DOMAINS=(`sqlite $TEMP_DB "SELECT COUNT (domain) from gravity_dup;"`)
                      ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                      ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
        LEFT_DOMAINS=($(sqlite $TEMP_DB "SELECT COUNT (domain) from gravity_dup;"))


In pihole_adlist_tool line 855:
    echo "  [i]  Enabling adlists with id ${adlist_conf_minimal_enabled[@]}"
                                          ^-- SC2145: Argument mixes string and array. Use * or separate argument.


In pihole_adlist_tool line 871:
            docker exec $CONTAINER_ID pihole restartdns reload-lists
                        ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
            docker exec "$CONTAINER_ID" pihole restartdns reload-lists


In pihole_adlist_tool line 908:
            docker exec $CONTAINER_ID pihole restartdns reload-lists
                        ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
            docker exec "$CONTAINER_ID" pihole restartdns reload-lists


In pihole_adlist_tool line 984:
        all_domains=(`sqlite $TEMP_DB "SELECT domain FROM all_domains"`)
                     ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                     ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
        all_domains=($(sqlite $TEMP_DB "SELECT domain FROM all_domains"))


In pihole_adlist_tool line 987:
                pihole-FTL regex-test $CURRENT_DOMAIN |grep -E -o "DB ID [0-9]*"|awk '{print $3}' | while read REGEX_ID; do
                                      ^-------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                          ^--^ SC2162: read without -r will mangle backslashes.

Did you mean:
                pihole-FTL regex-test "$CURRENT_DOMAIN" |grep -E -o "DB ID [0-9]*"|awk '{print $3}' | while read REGEX_ID; do


In pihole_adlist_tool line 1003:
        all_exact_domains=(`sqlite $GRAVITY "SELECT domain FROM domainlist WHERE type in (0,1);"`)
                           ^-- SC2207: Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                           ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.

Did you mean:
        all_exact_domains=($(sqlite $GRAVITY "SELECT domain FROM domainlist WHERE type in (0,1);"))


In pihole_adlist_tool line 1006:
          pihole-FTL regex-test $CURRENT_DOMAIN |grep -E -o "DB ID [0-9]*"|awk '{print $3}' | while read REGEX_ID; do
                                ^-------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                                    ^--^ SC2162: read without -r will mangle backslashes.

Did you mean:
          pihole-FTL regex-test "$CURRENT_DOMAIN" |grep -E -o "DB ID [0-9]*"|awk '{print $3}' | while read REGEX_ID; do


In pihole_adlist_tool line 1036:
        echo "  [i]  Since "$DATE_FIRST_ANALYZED" you have been visiting ${bold}"$NUM_ALL_DOMAINS" different domains${normal}."
                            ^------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                            ^------------------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                 ^--------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                 ^--------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "  [i]  Since ""$DATE_FIRST_ANALYZED"" you have been visiting ${bold}""$NUM_ALL_DOMAINS"" different domains${normal}."


In pihole_adlist_tool line 1037:
        echo "       You have ${bold}"$NUM_REGEX" blacklist RegEx${normal} configured ("$NUM_ENABLED_REGEX" enabled)"
                                      ^--------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                      ^--------^ SC2086: Double quote to prevent globbing and word splitting.
                                                                                        ^----------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                        ^----------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "       You have ${bold}""$NUM_REGEX"" blacklist RegEx${normal} configured (""$NUM_ENABLED_REGEX"" enabled)"


In pihole_adlist_tool line 1038:
        echo "       With your enabled blacklist RegEx you would have covered ${bold}"$NUM_DOMAINS_BLOCKED_BY_REGEX" different domains${normal}."
                                                                                      ^---------------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                                                                      ^---------------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
        echo "       With your enabled blacklist RegEx you would have covered ${bold}""$NUM_DOMAINS_BLOCKED_BY_REGEX"" different domains${normal}."


In pihole_adlist_tool line 1055:
            echo "       ${bold}"$NUM_DOMAINLIST_REGEX" distinct domain(s)${normal} from your domainlist are covered by your RegEx."
                                 ^-------------------^ SC2027: The surrounding quotes actually unquote this. Remove or escape them.
                                 ^-------------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
            echo "       ${bold}""$NUM_DOMAINLIST_REGEX"" distinct domain(s)${normal} from your domainlist are covered by your RegEx."

For more information:
  https://www.shellcheck.net/wiki/SC2068 -- Double quote array expansions to ...
  https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ...
  https://www.shellcheck.net/wiki/SC2027 -- The surrounding quotes actually u...

If I find the time, I will shellcheck this and open a PR… But if another person has the time and fun to shellcheck this - let's go, and leave an update here! πŸ‘πŸ»

Question

HI
Please can you tell me how i install and run this?

pihole_adlist_tool .. start probleme ^^

Bin also wegen der Installation Routine hier ..

Und frage mich was soll passieren nachdem ich starten will..

pi@ras:~ $ chmod +x ./pihole_adlist_tool
pi@ras:~ $
pi@ras:~ $ ./pihole_adlist_tool
-bash: ./pihole_adlist_tool: Is a directory
pi@ras:~ $ /pihole_adlist_tool
-bash: /pihole_adlist_tool: No such file or directory

https://ibb.co/wRzCwzz

Wrong detection of PIHOLE_DOCKER?!

I executed docker exec -it pihole bash and then:

root@pihole-merz-nimbus:/etc/pihole# ./pihole_adlist_tool
  *** Pihole Adlist Tool 2.6 ***
   ++++++++ Info ++++++++
  [i]  PIHOLE_DOCKER: No
  [i]  PIHOLE_DNSMASQ_VERSION: 2.87test4-18
  [i]  SQLITE_VERSION: 3.37.1
  [i]  DAYS_REQUESTED: 30
  [i]  TOP: Not shown
  [i]  SORT_ORDER: id ASC
  [i]  UNIQUE: Not shown
  [i]  REGEX_MODE: Disabled
  [i]  There is an update available: 2.6.0
…

So I can say for sure that this Pi-Hole is running within Docker πŸ˜† But your really nice script is detecting this wrong/not.

Also the update-detection is not correct - or version of 2.6 is set wrong and should be 2.6.0!?

DB Access denied

~/pihole_adlist_tool$ sudo ./pihole_adlist_tool

*** Pihole Adlist Tool ***

++++++++ Info ++++++++

[i] DAYS_REQUESTED: 30
[i] TOP: Not shown
[i] SORT_ORDER: id ASC
[i] UNIQUE: Not shown
[i] NEW_ADLIST_FILENAME_SCHEMA: no
Error: unable to open database "file:/etc/pihole/pihole-FTL.db?mode=ro": unable to open database file
[i] CNAME_AVAILABLE: deep CNAME info not available
[i] SQLITE_VERSION: 3.31.1

++++++++++++++++++++++

Error: unable to open database "file:/etc/pihole/pihole-FTL.db?mode=ro": unable to open database file
Error: unable to open database "file:/etc/pihole/pihole-FTL.db?mode=ro": unable to open database file
Error: unable to open database "file:/etc/pihole/pihole-FTL.db?mode=ro": unable to open database file
Error: unable to open database "file:/etc/pihole/pihole-FTL.db?mode=ro": unable to open database file

[i] Warning: You requested to analyze the last 30 day(s) (starting from 28.11.2020 3:11:16),
but last query is from 01.01.1970 0:00:00
[i] Nothing to do here. Exiting

Docker compability

The newest version doesn't work with docker anymore...

Can you add docker support again? Thank you!

Parse error near line 14: no such column: additional_info

I downloaded 2.6.4, but it contains 2.6.3

And it error-ed out as below:

$./pihole_adlist_tool -d 0 -s total

*** Pihole Adlist Tool 2.6.3 ***

++++++++ Info ++++++++
[i] PIHOLE_DOCKER: No
[i] PIHOLE_DNSMASQ_VERSION: v2.89-9461807
[i] SQLITE_VERSION: 3.42.0
[i] DAYS_REQUESTED: all time
[i] TOP: Not shown
[i] SORT_ORDER: total_domains DESC
[i] UNIQUE: Not shown
[i] REGEX_MODE: Disabled
[i] There is an update available: 2.6.4
++++++++++++++++++++++

Would you like to analyze your current adlist configuration or first enable all adlists (current can be restored later)?

  1. Current adlist configuration
  2. Enable all adlists (runs pihole -g)

Please select: 1

[i] Keeping current adlist configuration

[i] Calculating.....
[i] This might take some time - please be patient.
Parse error near line 14: no such column: additional_info
INSERT INTO cname(additional_info, hits) SELECT additional_info, COUNT(domain)
error here ---^

`

Errors encountered

Thanks for the nice tool! It runs for me, but appears to encounter a few issues. Here's a portion of the output --

  [i]  Calculating.....
  [i]  This might take some time - please be patient.
Error: near line 10: no such column: COUNT
Error: near line 11: no such column: COUNT
Error: near line 12: no such column: COUNT

  [i]  You have 9 adlists configured (5 enabled)
  [i]  Your gravity.db contains 157966 unique domains
  [i]  Since 22.12.2020 10:25:22 1108 different domains from your adlists have been blocked 336922 times in total
       (blocked directly by gravity or during deep CNAME inspection)
  [i]  Using you current adlist configuration 1316 domains would have been blocked 440581 times


  Press enter to continue...
./pihole_adlist_tool: line 557: [: : integer expression expected
./pihole_adlist_tool: line 577: [: : integer expression expected

Pi-hole is running on a Ubuntu 20.04 VM and was just updated today to the latest version ( Pi-hole v5.2.4 Web Interface v5.3.1 FTL v5.5.1)

Any idea on why these errors are occurring?

ABP style adlists

Report currently does not show the number for ABP style adlists.
Please consider include calculation for ABP style adlists.

Thank you

  [i]  Adlist coverage

id   enabled  total_domains  domains_covered  hits_covered  unique_domains_covered  address
---  -------  -------------  ---------------  ------------  ----------------------  ---------------------------------------------------------------------------------
413  1        0                                                                     https://big.oisd.nl/
469  1        0                                                                     https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/multi.txt
470  1        0                                                                     https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/tif.txt

Strange analysis result

I am using the tool to optimize my adlists. Currently I am adding lists from https://github.com/blocklistproject, but the analysis results are not plausible and not stable.

  1. initial run
$ pihole -g && ./pihole_adlist_tool 
(...)
id          enabled     total_domains  domains_covered  hits_covered  unique_domains_covered  address                                                         
----------  ----------  -------------  ---------------  ------------  ----------------------  ----------------------------------------------------------------
1           0                                                                                 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
2           0                                                                                 https://mirror1.malwaredomains.com/files/justdomains            
3           1           20567          108              27820                                 http://sysctl.org/cameleon/hosts                                
5           0                                                                                 https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
6           0                                                                                 https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt      
8           1           23660          16               32            16                      https://raw.githubusercontent.com/kboghdady/youTube_ads_4_pi-hol
9           1           156039         468              66207         309                     https://blocklistproject.github.io/Lists/ads.txt                
10          1           1220                                                                  https://blocklistproject.github.io/Lists/abuse.txt   
  1. disable http://sysctl.org/cameleon/hosts
$ pihole -g && ./pihole_adlist_tool 
(...)
id          enabled     total_domains  domains_covered  hits_covered  unique_domains_covered  address                                                         
----------  ----------  -------------  ---------------  ------------  ----------------------  ----------------------------------------------------------------
1           0                                                                                 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
2           0                                                                                 https://mirror1.malwaredomains.com/files/justdomains            
3           0                                                                                 http://sysctl.org/cameleon/hosts                                
5           0                                                                                 https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
6           0                                                                                 https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt      
8           1           23660          16               32            16                      https://raw.githubusercontent.com/kboghdady/youTube_ads_4_pi-hol
9           1           156039         468              66212                                 https://blocklistproject.github.io/Lists/ads.txt                
10          1           156039         468              66212                                 https://blocklistproject.github.io/Lists/abuse.txt              

-> Unexpectedly, the number of total domains for https://blocklistproject.github.io/Lists/abuse.txt has increased from 1220 to 156039! And why do ads.txt and abuse.txt show identical numbers?

Error: stepping, database is locked

Any ideas?


  [i]  Domains from disabled adlists are not stored in gravity's database.
       If you want to include them in the analysis, run this script again and select 'Enable all adlists'
       As the same domains usually appears on more than one adlist the sum of covered domains from this table is greater 
       than the number of calculated blocked domains shown above


  [i]  In total your adlists contain 190 visited (covered) unique domains - meaning those domains are contained only in a single adlist. 


  Would you like to ... 

  1)  Keep your current adlist configuration
  2)  Enable only adlists with covered unique domains
  3)  Enable the minimal number of adlists that cover all domains that would have been blocked

  Please select: 3

  [i]  Enabling minimum number of adlists that cover all domains that would have been blocked....
Error: stepping, database is locked (5)

  [i]  Enabling adlists with id 56 58 60 66 67 74 81 82 83 85 90 102 104 108
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
Error: stepping, database is locked (5)
  [βœ“] Reloading DNS lists

  [βœ“]  Minimal number of adlists that cover all domains (that would have been blocked) enabled


  [i]  Of the 704 domains that would have been blocked by the adlist configuration you chose to analyze,
       704 domains are covered by your current adlist configuration.

  [i]  Removing temporary database...
  [βœ“]  Temporary database removed

Can't read /etc/pihole/pihole-FTL.db

When pihole_adlist_tool I get the following output:

~/pihole_adlist_tool (master) --ϐ ./pihole_adlist_tool                                                                                   


  *** Pihole Adlist Tool 2.6.3 ***


   ++++++++ Info ++++++++

  [i]  PIHOLE_DOCKER: No
  [i]  PIHOLE_DNSMASQ_VERSION: v2.87
  [i]  SQLITE_VERSION: 3.39.2
  [i]  DAYS_REQUESTED: 30
  [i]  TOP: Not shown
  [i]  SORT_ORDER: id ASC
  [i]  UNIQUE: Not shown
  [i]  REGEX_MODE: Disabled

   ++++++++++++++++++++++



  [i]  Warning: You requested to analyze the last 30 day(s) (starting from 14.09.2022	19:56:48),
       but last query is from 31.12.1969	17:00:00
  [i]  Nothing to do here. Exiting 

Running

pihole-FTL sqlite3 "file:/etc/pihole/pihole-FTL.db" "SELECT MIN(timestamp) FROM queries;"

or

pihole-FTL sqlite3 "file:/etc/pihole/pihole-FTL.db" "SELECT MAX(timestamp) FROM queries;"

by itself returns nothing, so I believe that there's an issue with the command getting data from the db.

Output of pihole_adlist_tool -v

~/pihole_adlist_tool (master) --ϐ ./pihole_adlist_tool -v                                                                                
Pihole Adlist Tool v2.6.3

[Question] If a domain exists in >=2 adlists, each of which have no unique domains, will the domain no longer be blocked (if told to remove adlists without unique ones)?

The question in the title may be phrased a little oddly, but I'll try to explain it better here. Take the following example scenario:

I have 5 adlists. For simplicity's sake, I'll just reference them by their id 1-5.

Let's use the following data for these 5 adlists (the specific numbers don't really matter, just simply whether a domain/unique domain is present or not):

id domains_covered unique_domains_covered
1 24 3
2 43 6
3 1
4 1
5

Now, let's say I choose to "Enable only adlists with covered unique domains" on the final question. From my understanding, only adlists 1 and 2 will remain enabled, while the rest will be disabled.

Notice how lists 3 and 4 have one covered domain. Let's say this domain is example.com. example.com exists in both list 3 and 4, but not any of the other lists. I assume that this would mean that neither list 3 nor list 4 are considered to have a unique covered domain.

Would it be correct to say that, if you were to let the script enable only adlists containing unique domains, that protection against the example.com domain would be lost? Or is there something in the script that accounts for this that I'm missing? I feel that for people who use a large amount of adlists, this could be a fairly common "special case."

More importantly, if example.com does end up being lost, is there an easy way to determine a list of domains like this one that would be lost by skimming the adlists, in a real scenario where you don't know the all the specific domains in each adlist?

Does regex checking account for Filter Lists?

When using the regex checking feature, does the list of blocked domains include the fact that they might also be blocked by a filter list, or is the regex checker independent of the filter list checker?

I am trying to determine if I need the regex configurations that I have applied.

Instructions

Hello,

Would you kindly provide instructions for installation and execution of script in the README? Thanks!

Line 7 problem

Hello, After downloading the file, here is what I have:
image

Any Idea?
Thanks

Add assigned group(s) for adlist in result

It would really be nice to have the assigned group for each adlist also in the resulting table in addition to "id, enabled, total_domains, domains_covered, hits_covered, unique_domains_covered, address" which should also be sortable:

# sqlite3 ./gravity.db "select * from 'group'"
0|1|Default|1604414479|1614120328|The default group
22|1|adult-content|1614119689|1614120454|
23|1|test-group|1637245963|1637245963|for testing before going in production

# sqlite3 ./gravity.db "select * from adlist_by_group order by 1"
1|0
5|0
13|0
19|0
20|0
21|0
23|0
27|0
28|0
30|0
32|0
33|0
34|0
35|0
37|0
42|22
43|0
59|0
60|0
62|0
63|0
64|0
67|0
68|0
69|0
71|22
72|22
73|22
74|0
77|22
80|22
81|22
82|22
82|23
83|0
84|0
85|0
87|0
88|22
89|0
90|0

If I find some time, I will make a PR and tell you when I'm starting before you're also trying to add this enhancement.

Running tool outside the Docker-container throws some errors

I was running it from within the docker-container and from outside the container. From outside it throws some errors:

  Would you like to ...

  1)  Keep your current adlist configuration
  2)  Enable only adlists with covered unique domains
  3)  Enable the minimal number of adlists that cover all domains that would have been blocked

  Please select: 2

  [i]  Enabling adlists with covered unique domains....
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here
Error: in prepare, unrecognized token: "'"
  '
  ^--- error here

  [βœ“]  Adlists with covered unique domains enabled


  [i]  Of the 713 domains that would have been blocked by the adlist configuration you chose to analyze,
       711 domains are covered by your current adlist configuration.

  [i]  Removing temporary database...
  [βœ“]  Temporary database removed

I like the tool. I have about 40 adlists and after the check only 12 need to be enabled.

Receiving an error "database disk image is malformed"

Hi

I recently installed this tool to check the efficiency of the adlists I have on my PiHole. When I ran the tool for the first time, this is the output:

*** Pihole Adlist Tool ***

++++++++ Info ++++++++

[i] PIHOLE_DNSMASQ_VERSION: 2.84
[i] SQLITE_VERSION: 3.34.0
[i] DAYS_REQUESTED: 30
[i] TOP: Not shown
[i] SORT_ORDER: id ASC
[i] UNIQUE: Not shown
[i] REGEX_MODE: Disabled

++++++++++++++++++++++

Error: database disk image is malformed
Would you like to analyze your current adlist configuration or first enable all adlists (current can be restored later)?

  1. Current adlist configuration
  2. Enable all adlists (runs pihole -g)

Please select: 1

[i] Keeping current adlist configuration

[i] There is a mismatch between your enabled adlists and the data found in the gravity database.
You have 47 adlists enabled, but data from 46 (patially different) adlists in your gravity database.
You're likely disabled/enabled adlist without running gravity afterwards.
It's highly recommended to run gravity now to solve the differences, otherwise this tool will analyze the available data.

Would you like to run gravity now?

  1. Yes
  2. No

Please select: 2

[i] Not running gravity, keeping mismatch between enabled adlists and the data found in the gravity database.

[i] Calculating.....
[i] This might take some time - please be patient.
Error: near line 4: database disk image is malformed
Error: near line 14: database disk image is malformed

Database locked error

So I ran the script using bash, and it said Error: database is locked. I think I'm running it wrong.

Please add a licence

There doesn't seem to be a licence attached to this repo. In this state no one would be allowed to publish a derivative work, or even copy the code for a personal project. This might be an intentional, but usually it's an oversight. Quick read up on licenses here.

List the domains which generated a hit

Hey,
Thanks so much for your tool, it allowed me to really streamline my lists.

I was wondering if you might be able to help with another issue. I am trying to figure out a way to see which domains have triggered in each list.
I have a malware list loaded and it is really difficult to figure out which domains are being triggered on my network. Using the pihole webgui there is no way to search for blocked domains by list.

I thought with some tweaking your tool might be able to do that?

This is my output when I run the tool with -a

`[i] Adlist coverage

id enabled total_domains domains_covered hits_covered unique_domains_covered address


1 1 209627 593 60232 589 https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
5 1 167912 1 2 https://v.firebog.net/hosts/RPiList-Malware.txt `

I want to know what the domain is that is listed against the malware list so I can search for it in the webgui to find out which client triggered the hit

Many thanks

Starting with version 2.4.0, you get multiple 'environment: line 5: docker: command not found' messages

Am still checking what's happening here, but it seems that somewhere in the script the docker command gets called when it shouldn't (and as end result no changes get applied).

Other info:

  • The script detects/sets PIHOLE_DOCKER to 0 (when it should).
  • To reproduce the bug run the script with 0 params and pick option 1(Current adlist configuration) followed by 3 (Enable the minimal number of adlists that cover all domains that would have been blocked).
  • I think this might be happening due to SUDO_SQLITE. Am assuming the value of PIHOLE_DOCKER doesnt stay/is visible when we call SUDO_SQLITE, but am really not sure.

Possibility to Deactivate redundant Blocklists

Hello,

is there a possibility to show and/or deaktivate blocklists that are redundant, so that the unique domains on the first page does not change and only the unnecessary blocklists with their domains are deactivated. Thanks in advance.

Instructions?

Could you please provide a step-by-step guide on how to install/use this tool?

Rgds
newbie

[Recommendation] - Make Current Adlist Default Option

Thank you for making this wonderful tool, I noticed that after you run the tool, an option relating to Analysis of Either All or Current Adlist comes up. This could either be (A) Defaults to 5 seconds to automatically select current or (B) be available only if Interactive mode has been used to run the script.

Use extended adlist info from gravity database

Note to myself:
Table adlist is updated with total number of domains for each adlist_id .
Since commit 73963fecda6dc65b10d1dd3e43a5931dc531304a to pihole's core, locally saved adlist copies contain the adlist_id in the filename. We use that to count the lines in each file and use the adlist_id to attribute it to the corresponding adlist in TEMP_DB This is faster than to count the domains for each adlist from gravity_db.

But: in a next Pi-hole version, Pi-hole hopefully will store the number of domains for each adlist as additional info. This can then be used. This saves time (no calculation has to be done again) and makes the script Independent of the locally saved adlist copies.

pi-hole/pi-hole#3951

"SUDO_SQLITE: command not found"

  1. Keep all adlists enabled
  2. Enable only adlists with covered unique domains
  3. Enable the minimal number of adlists that cover all domains that would have been blocked
  4. Restore previous adlist configuration

Please select: 2

[i] Enabling adlists with covered unique domains....
bash: 0=0: command not found
bash: SUDO_SQLITE: command not found
bash: sqlite: command not found
bash: 0=0: command not found
bash: SUDO_SQLITE: command not found
bash: sqlite: command not found
bash: 0=0: command not found
bash: SUDO_SQLITE: command not found
bash: sqlite: command not found
[βœ“] Reloading DNS lists

[βœ“] Adlists with covered unique domains enabled

[βœ“] Of the 833 domains that would have been blocked by the adlist configuration you chose to analyze,
0 domains are covered by your current adlist configuration.

[i] Removing temporary database...
[βœ“] Temporary database removed

Note that:
$ sqlite

-bash: sqlite: command not found

$ sqlite3

SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.

So I created an alias "sqlite='sqlite3'" but that did not solve the error message.
How would I be able to fix this?

I'm also a bit worries about the message:

[βœ“] Of the 833 domains that would have been blocked by the adlist configuration you chose to analyze,
0 domains are covered by your current adlist configuration.

Does this mean that there's domains not covered (833) that should be covered?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.