Comments (16)
Case in point why we should not be modifying a distributed file:
Switching clone ./src/urlextract to git://github.com/rickmeasham/URLExtract.git (to revision master)
Running command git config remote.origin.url git://github.com/rickmeasham/URLExtract.git
Running command git checkout -q master
error: Your local changes to the following files would be overwritten by checkout:
urlextract/data/tlds-alpha-by-domain.txt
Please commit your changes or stash them before you switch branches.
from urlextract.
Hi, thanks for reporting this issue. OK there is option for setting up this directory when you are using the class it self within your python code. I missed this and forget to add this feature as parameter to the urlextract
command.
Thanks!
from urlextract.
@lipoja do you mind sharing how to do it in python code as I saw in code URLExtract takes a CacheFile type argument
from urlextract.
It can be set during object initialization: extractor = URLExtract(cache_dir='/path/to/cache')
from urlextract.
I am developing a tool, I use pyinstaller, with the noconsole option and it gives cache problems.
I use "extractor = URLExtract(cache_dir=''D:\temp')"
but it seems like no such path is established
from urlextract.
@B16f00t Does it crash? Is there any error message shown? Or it just does not work at all?
from urlextract.
My code:
from urlextract import URLExtract
try:
extractor = URLExtract(cache_dir="D:\\")
urls = extractor.find_urls(text)
print(urls)
except Exception as e:
print(e)
ERROR:
Default cache file does not exist 'C:\Users\B16f00t\AppData\Local\Temp_MEI102122\urlextract\data\tlds-alpha-by-domain.txt'!
from urlextract.
@B16f00t Turn on logging and see if there are any other messages. It looks like the path "D:\" does not exist, or is not writable for the user.
Let me know if it logs some other errors (or info messages), thanks.
from urlextract.
Hi @lipoja, I have the same issue. The script when uploaded as AWS lambda, the script fails.
I init the UrlExtract with a cache_dir. However, we get the error log "None of cache directories is writable."
It seems we set the "cache_dir" in the initialization step. But the "cache_dir" wasn't given in _get_cache_lock_file_path()
.
For example,
def _load_cached_tlds(self):
"""
Loads TLDs from cached file to set.
:return: Set of current TLDs
:rtype: set
"""
# check if cached file is readable
if not os.access(self._tld_list_path, os.R_OK):
self._logger.error("Cached file is not readable for current "
"user. ({})".format(self._tld_list_path))
raise CacheFileError(
"Cached file is not readable for current user."
)
set_of_tlds = set()
with filelock.FileLock(self._get_cache_lock_file_path()):
with filelock.FileLock(self._get_cache_lock_file_path()):
should be with filelock.FileLock(self._get_cache_lock_file_path(cache_dir)):
.
line 204 may has the same problem as well with filelock.FileLock(self._get_cache_lock_file_path()):
.
Thank you.
from urlextract.
Hi @lipoja, is there a fix for this yet? There's a fix_cache
branch that appears to be aimed at fixing this issue.
Below are my observations before I saw that branch ...
There appear to be two intermingled issues here:
- Setting
cache_dir
is only used for the cache file and not the lock file - When using the default list provided with the distribution, we still attempt to lock the file. And because the
cache_dir
isn't used for the lock file, we cannot get that lock
I was going to suggest that we shouldn't bother getting a lock on the distributed file, but I see that you allow the module to update the distributed file. I feel like that's an anti-pattern and we should have a cache file somewhere in user-space (or even /tmp if we can't write somewhere more permanent). The default file should be a read-only failover.
Unless that is changed, then the only solution is to create a lock file somewhere we are (reasonably) always allowed to write. Given it's a lock file and thus very ephemeral we could just write that to /tmp or the OS equivalent. Or we could allow the user to specify a lock_file
in the constructor.
from urlextract.
I've grabbed the fix_cache
branch, but when the cache file is not available in the cache_dir
, calling update()
attempts to update the default file. To fix this for now, I'm generating an empty file before initialising the extractor, then calling update if the file is empty. This hack works even when the module is in a Lambda layer.
cache_file = Path('/tmp/tlds-alpha-by-domain.txt')
if not cache_file.exists():
cache_file.touch()
extractor = URLExtract(cache_dir='/tmp')
if cache_file.stat().st_size == 0:
extractor.update()
from urlextract.
Any update on this bug? Would love the use this in AWS. Are folks just using a package from the fix_cache branch with Ricks work-around? Is there a way to PIP install that branch?
from urlextract.
@paulfdietrich said:
Is there a way to PIP install that branch?
You can use this, but I'm not going to be tracking any changes to the official repo. Caveat emptor.
pip install https://github.com/rickmeasham/URLExtract/archive/master.zip#egg=urlextract
from urlextract.
Hi, thank you both for your patience and also for time spend on reporting this issue.
I've modified the code in fix_cache
branch, could you check if it is working for you? It is getting late and do not have "the non working" environment handy.
I kept the solution that was in the fix_cache
branch and I've tried to fix the issue with non existing file.
Thank you!
from urlextract.
Case in point why we should not be modifying a distributed file:
Switching clone ./src/urlextract to git://github.com/rickmeasham/URLExtract.git (to revision master) Running command git config remote.origin.url git://github.com/rickmeasham/URLExtract.git Running command git checkout -q master error: Your local changes to the following files would be overwritten by checkout: urlextract/data/tlds-alpha-by-domain.txt Please commit your changes or stash them before you switch branches.
I think this can be solved by adding tlds-alpha-by-domain.txt to .gitignore. I did that already so it should be fine once your .gitignore is updated from latest master branch.
from urlextract.
This fix is released on pypi. If somebody has a chance to run it on aws it would be graet.
Thanks
from urlextract.
Related Issues (20)
- IPv6? HOT 1
- left walk does not stop on various unicode chars HOT 1
- should not grab email fragments HOT 1
- comma extracted at the end if url ends with comma HOT 3
- travis-ci seems no longer active repository HOT 2
- URLExtract() init really slow
- ERROR: Can not download list of TLDs. (URLError: [Errno 104] Connection reset by peer) HOT 2
- Does Not extract the URL that is leading special character
- urlextract without authority causes AttributeError HOT 2
- Wrong indices and incomplete extraction when string contains similar urls HOT 1
- Handle upper-case false positives HOT 9
- Unable to detect t.me links HOT 1
- Bug with flag `allow_mixed_case_hostname=False` HOT 4
- Extracting Markdown Text, doesn't process escaped \\ correctly HOT 1
- Support non-unicode hostname HOT 3
- Support for private/reserved/custom TLDs
- Wrong indices and repeated matches when hostname contains the TLD
- Invalid URLs accepted with subdomains
- red flag from antiviruses HOT 2
- [Errno 11002] Temporary failure in name resolution after using URLExtract HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from urlextract.