Url-Guarding Issue

Discuss and get help configuring CacheGuard URL blacklist Guarding
axnav
Posts: 12
Joined: 08 Jan 2017 08:11

Url-Guarding Issue

Post by axnav » 11 May 2017 12:10

Hi David,

I implemented our cacheguard appliance as a non transparent proxy.
The http and https traffic from our clients flows over the cacheguard --> works fine!

I activated the automatic download of the blacklists from your ftp Server --> works fine!

Autoupdate works fine too:

2017/05/11-03:28:03 Begining the URL list update
2017/05/11-03:28:03 Loading Advert domains contents
2017/05/11-03:28:06 Loading Advert urls contents
2017/05/11-03:28:06 Loading Chat domains contents
2017/05/11-03:28:10 Loading Chat urls contents
2017/05/11-03:28:10 Loading Dating domains contents
2017/05/11-03:28:13 Loading Dating urls contents
2017/05/11-03:28:13 Loading Drugs domains contents
2017/05/11-03:28:16 Loading Drugs urls contents
2017/05/11-03:28:16 Loading Porn domains contents
2017/05/11-03:28:19 Loading Porn urls contents
2017/05/11-03:28:19 Loading Redirector domains contents
2017/05/11-03:28:23 Loading Redirector urls contents
2017/05/11-03:28:23 Loading SpyWare domains contents
2017/05/11-03:28:26 Loading SpyWare urls contents
2017/05/11-03:28:26 Loading Violence domains contents
2017/05/11-03:28:29 Loading Violence urls contents
2017/05/11-03:28:29 Updating (1) the 'Advert(domains)' list
2017/05/11-03:28:29 Updating (1) the 'Advert(urls)' list
2017/05/11-03:28:29 Updating (1) the 'Chat(domains)' list
2017/05/11-03:28:29 Updating (1) the 'Chat(urls)' list
2017/05/11-03:28:29 Updating (1) the 'Dating(domains)' list
2017/05/11-03:28:29 Updating (1) the 'Dating(urls)' list
2017/05/11-03:28:29 Updating (1) the 'Drugs(domains)' list
2017/05/11-03:28:29 Updating (1) the 'Drugs(urls)' list
2017/05/11-03:28:29 Updating (1) the 'Porn(domains)' list
2017/05/11-03:28:29 Updating (1) the 'Porn(urls)' list
2017/05/11-03:28:29 Updating (1) the 'Redirector(domains)' list
2017/05/11-03:28:29 Updating (1) the 'Redirector(urls)' list
2017/05/11-03:28:29 Updating (1) the 'SpyWare(domains)' list
2017/05/11-03:28:29 Updating (1) the 'SpyWare(urls)' list
2017/05/11-03:28:29 Updating (1) the 'Violence(domains)' list
2017/05/11-03:28:29 Updating (1) the 'Violence(urls)' list
2017/05/11-03:28:30 Refreshing the URL Guard
2017/05/11-03:28:30 Ending the URL list update



Then I created the guard filter for the IP range and after this step I created the guard policy only with the IP filter.
Finally I definded the guard rule.

But Cacheguard wont block the traffic to the blacklisted URLs/Domains for example youporn.com

Here are the Output from console which mybe helpful:


login as: admin
Welcome to the CacheGuard Web Gateway
Management URL: https://xxxx:8090 from allowed networks
Copyrights (C) 2002-2016 CacheGuard - All rights reserved

admin@xxx.xxx.xxx.xxx's password:
Access denied
admin@xxx.xxx.xxx.xxx's password:
admin@cacheguard> mode guard

mode guard on

admin@cacheguard> mode transparent

mode tweb on

admin@cacheguard> mode sslmediate

mode sslmediate off

admin@cacheguard> urllist

urllist Advert
Chat
Dating
Drugs
Porn
Redirector
SpyWare
Violence

admin@cacheguard> guard filter

guard filter ip FilterExtWlanRange network 172.x.x.x/255.255.0.0
guard filter time <null>
guard filter ldap <null>


admin@cacheguard> guard policy

guard policy myNAVAXExtWlanPolicy: ip FilterExtWlanRange

admin@cacheguard> guard rule

guard rule default deny: Porn
myNAVAXExtWlanPolicy deny: Porn
Advert
Violence
SpyWare
Redirector
Drugs
Dating
Chat

admin@cacheguard>


What I did wrong ?

Kindly regards,

michael
Last edited by axnav on 11 May 2017 14:15, edited 1 time in total.

User avatar
david
Posts: 154
Joined: 08 Aug 2015 20:38

Re: Url-Guarding Issue

Post by david » 11 May 2017 12:58

Hi Michael,

Your configuration seems good. Can you please send us the output of the following commands:

Code: Select all

urllist report
urllist auto
Best Regards,
David Jan
CacheGuard Technical Team
https://www.cacheguard.com

axnav
Posts: 12
Joined: 08 Jan 2017 08:11

Re: Url-Guarding Issue

Post by axnav » 11 May 2017 14:14

Hi David,

thank you a lot for your quick response, here is the requested output:

login as: admin
Welcome to the CacheGuard Web Gateway
Management URL: https://xxx.xxx.xxx.xxx:8090 from allowed networks
Copyrights (C) 2002-2016 CacheGuard - All rights reserved

admin@xxx.xxx.xxx.xxx's password:
admin@cacheguard> urllist report

2017/05/11-03:28:03 Begining the URL list update [ OK ]
2017/05/11-03:28:03 Loading Advert domains contents [ OK ]
2017/05/11-03:28:06 Loading Advert urls contents [ OK ]
2017/05/11-03:28:06 Loading Chat domains contents [ OK ]
2017/05/11-03:28:10 Loading Chat urls contents [ OK ]
2017/05/11-03:28:10 Loading Dating domains contents [ OK ]
2017/05/11-03:28:13 Loading Dating urls contents [ OK ]
2017/05/11-03:28:13 Loading Drugs domains contents [ OK ]
2017/05/11-03:28:16 Loading Drugs urls contents [ OK ]
2017/05/11-03:28:16 Loading Porn domains contents [ OK ]
2017/05/11-03:28:19 Loading Porn urls contents [ OK ]
2017/05/11-03:28:19 Loading Redirector domains contents [ OK ]
2017/05/11-03:28:23 Loading Redirector urls contents [ OK ]
2017/05/11-03:28:23 Loading SpyWare domains contents [ OK ]
2017/05/11-03:28:26 Loading SpyWare urls contents [ OK ]
2017/05/11-03:28:26 Loading Violence domains contents [ OK ]
2017/05/11-03:28:29 Loading Violence urls contents [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Advert(domains)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Advert(urls)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Chat(domains)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Chat(urls)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Dating(domains)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Dating(urls)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Drugs(domains)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Drugs(urls)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Porn(domains)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Porn(urls)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Redirector(domains)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Redirector(urls)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'SpyWare(domains)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'SpyWare(urls)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Violence(domains)' list [ OK ]
2017/05/11-03:28:29 Updating (1) the 'Violence(urls)' list [ OK ]
2017/05/11-03:28:30 Refreshing the URL Guard [ OK ]
2017/05/11-03:28:30 Ending the URL list update [ OK ]

admin@cacheguard> urllist auto

urllist auto Advert on load update daily ftp ftp.cacheguard.net Advert
Chat on load update daily ftp ftp.cacheguard.net Chat
Dating on load update daily ftp ftp.cacheguard.net Dating
Drugs on load update daily ftp ftp.cacheguard.net Drugs
Porn on load update daily ftp ftp.cacheguard.net Porn
Redirector on load update daily ftp ftp.cacheguard.net Redirector
SpyWare on load update daily ftp ftp.cacheguard.net SpyWare
Violence on load update daily ftp ftp.cacheguard.net Violence

admin@cacheguard>


Kindly regards,

Michael

User avatar
david
Posts: 154
Joined: 08 Aug 2015 20:38

Re: Url-Guarding Issue

Post by david » 11 May 2017 14:45

You are very welcome Michael.

It is possible that your URL lists have not been properly built. To download and build URL lists from scratch, please proceed as follows from the Web GUI:

1 - Go to the [SECURITY] > [URL & Domain Lists] > [Auto Load Contents] menu option.
2 - Select "Create" for all categories (you can just click on the "Operation" button at the top) and SUBMIT.
3 - Click on the blinking blue down arrow icon (in the mini menu bar) and SUBMIT.
4 - Go to [GENERAL] > [System Operations] > [Update URL Lists] menu option and SUBMIT.

Now you can make a test to see if denied URLs are properly blocked. Afterwards you can reconfigure the automatic updating to download updates only (and not complete lists). To do so, please proceed as follows:

5 - Return to the [SECURITY] > [URL & Domain Lists] > [Auto Load Contents] menu option.
6 - Select "Update" for all categories (you can just click on the "Operation" button) and SUBMIT.
7 - Click on the blinking blue down arrow icon (in the mini menu bar) and SUBMIT.

Please let me know if this procedure fixed the issue.

Best Regards,
David Jan
CacheGuard Technical Team
https://www.cacheguard.com

axnav
Posts: 12
Joined: 08 Jan 2017 08:11

Re: Url-Guarding Issue

Post by axnav » 11 May 2017 15:58

Hi David,

I did your steps and it worked immediatly^^

Thank you for your quick help.


Kindly regards,

michael

User avatar
david
Posts: 154
Joined: 08 Aug 2015 20:38

Re: Url-Guarding Issue

Post by david » 12 May 2017 09:14

Hi Michael,

I am happy to hear that and thank you for your feedback. However we didn't really identify the root cause of this issue. You are not the first user to encounter such an issue and it seems that there is a furtive anomaly in the CG-NG 1.2.6 that avoids to properly download URL lists the first time. This happen only in certain circumstances.

We are working on a new release (v1.3.0) that among other new features (such as Kerberos authentication and syslog) , reinforces the URL lists download process.

Best Regards,
David Jan
CacheGuard Technical Team
https://www.cacheguard.com

User avatar
charles
Site Admin
Posts: 41
Joined: 06 Nov 2014 16:23
Location: Paris
Contact:

Re: Url-Guarding Issue

Post by charles » 25 May 2017 16:06

Dear Michael,

The root cause that David is talking about could be the change of the download server (IP address or name) in your auto update/create configuration. Can you please tell us if prior to update/create from ftp.cacheguard.net you used another server (your own server) to initialize your blacklists?

Indeed, if you have this:

Code: Select all

urllist auto Advert on load update daily tftp 10.0.10.1 Advert
and if your first download is done from 10.0.10.1, changing the configuration to:

Code: Select all

urllist auto Advert on load update daily ftp ftp.cacheguard.net Advert
would have the effect to produce the inconsistency you have encountered. Do you remember if you had such a configuration change?

Your feedback would be greatly appreciated in helping us improving CacheGuard-OS.

Kind Regards,
Charles Tajvidi
IT Technical Architect
http://www.cacheguard.com

axnav
Posts: 12
Joined: 08 Jan 2017 08:11

Re: Url-Guarding Issue

Post by axnav » 02 Jul 2017 11:03

Dear Charles,

Sorry for my late answer.
No, with own blacklists loading from own server I started after solving this issue.

I found two new little "bug/issue" , I created an own simple domain blacklist.
And I used the same sftp account like I used it for backup or creating log exports.

At this moment I exported a log or created a backup and forget to move the file
away from the folder the auto update for this blacklist became broken.

In the GUI it shows only the red circle and the percentage stays at 7% or 97%.

I solved it with creating an own account for loading the selfcreated blacklist.

Please supply an howto in the documentation for creating own blacklists.

Here I found the second annoying bug:

Cacheguard cant handle two accounts to the same server IP, I had to add a second IP to the sftp servers interface.
If you try to create a second account it overwrites the first one.

Kindly regards,

Michael

PS: I upgraded (reinstalled) our cacheguard appliance from 1.26 to 1.31 and it worked fine.
I was a little confused about the re-registration process, but I got it finally :-)

axnav
Posts: 12
Joined: 08 Jan 2017 08:11

Re: Url-Guarding Issue

Post by axnav » 03 Jul 2017 10:35

Hi,

Something went wrong with the blacklists.

Yesterday I installed Cacheguard 1.3.1 and activated the blacklist supplied by cacheguard and my single blacklist.
In the autoload configuration I set preiodicity daily.

Then I started AutoUpdate Now and after few seconds every List was finished and marked green.

For the next 24 hours the blacklists worked fine, my own blacklist too.

Today morning I saw this the dashboard:

2017/07/03-12:16:46 Begining the URL list update
2017/07/03-12:16:46 Loading AddDomains domains contents (28) <--- marked with red circle
2017/07/03-12:16:53 Loading AddDomains urls contents (78) <--- marked with red circle
2017/07/03-12:16:53 Loading Advert domains contents
2017/07/03-12:16:54 Loading Advert urls contents
2017/07/03-12:16:54 Loading AnonymousVPN domains contents
2017/07/03-12:16:54 Loading AnonymousVPN urls contents

My own blacklists "AddDomains" are marked with red circle and
NONE of the blacklists works.
for example: Urls which was blocked by Warez blacklist yesterday, are unblocked this morning!

I had to switch all blacklists to Create --> submit --> switch to update -> submit --> AutoUpdate Now and then ALL blacklists including mine works again.


I added my self created blacklist files.

Maybe you can reproduce the issue.

Kindly regards,

Michael
Attachments
AddDomains.domains.gz
(109 Bytes) Downloaded 88 times
AddDomains.urls.gz
(50 Bytes) Downloaded 142 times

axnav
Posts: 12
Joined: 08 Jan 2017 08:11

Re: Url-Guarding Issue

Post by axnav » 03 Jul 2017 11:03

Hi,

interessting behaviour:
After 15 minutes of working blacklists , the blacklists stopped working again.

Now I deleted my blacklist and observe the behaviour.

Kindly regards,

Michael

Post Reply