yacodi / sitemap-generators Goto Github PK

Hi There,

I downloaded the latest version of sitemap generator (1.5) from your
download location yesterday. And I found an error with the sitemap_gen.py
script line number 1802 where the variable sitemap_index_header is written
incorrectly like: "sitemap__index_header" (with two underscores) for which
it is throwing error while trying to create sitemap index file. So I though
if I will inform you then you can modify the code and place the correct
version of code to download so other users will not face the similar problem.

Thanks,
Chittaranjan Pattnaik
Mindfire Solutions
http://www.mindfiresolutions.com

Original feature listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1510515&group_id=137793
&atid=739386

If the file has permissions o-r, then Google isn't
likely to be able to read the file. I request some type
of permissions filter be added to allow automatic
exclusion of these files. Thank you.

Original feature listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1765876&group_id=137793
&atid=739386

Running this on a Windows server requires absurd measures to make filter
patterns work:

<filter action="drop" type="regexp"
pattern="/[sS][oO][mM][eE][dD][iI][rR]/" />

There should be case-insensitive comparisons to work around Microsoft's
decades old filename case-insensitivity bug.

[deleted issue]

What steps will reproduce the problem?
1. Run setup
2. Configure sites through admin site 
3. Submit maps through google webmaster tools

What is the expected output? What do you see instead?
Expected to have sitemaps list for each virtual host.
A sitemap with googles generator tag and no url listings, that
when submitted to google through webmaster tools isn't acceptable as
a sitemap listing at all.
What version of the product are you using? On what operating system?
win-beta1.msi on 2003 iis6


Please provide any additional information below.

Going to look into the generators setup directory and see if there are any
templaes that can be edited to correct xml policies. Then reset generator
to default settings through admin which I basically set up to ignore
/cgi-bin /images /admin* and iis directories /_*

Original feature listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1598569&group_id=137793
&atid=739386

According to this blog post, Yahoo and MSN have decided to use this sitemap
format going forward. Because of this the namespace of the generated
sitemaps needs to be changed to comply with the new standard Sitemap
Protocol.

http://googlewebmastercentral.blogspot.com/2006/11/joint-support-for-sitema
p-protocol.html

Original feature listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1214402&group_id=137793
&atid=739386

As of this release, there are three ways to generate a
sitemap: specifying urls, specifying paths, or using logs.

However, I would image that many administrators will
use these methods to mimic exactly what their
robots.txt file specifies. Why not make this easier?

Something like this:
<robots url="http://www.example.com/robots.txt"
path="/var/www/html" bot="googlebot />

url: Address of robots.txt.

path: Root path of the site.

bot: Interest was shown in other search engines using
this software. This attribute will allow the sitemap
generation to follow the rules for a certain bot.
Values would include a bot name, or "*" to follow all
rules, regardless of the bot they are meant for.

This would essentially mimic a directory element, and a
few filter elements based on the rules within robots.txt.

Details will have to be ironed out, taking into account
aliased directories that a bot would see, but not
visible on the file system.

Thus, creation of the sitemap will follow the same
rules a bot would.

I am unable to Generate Site-Map for my Website
http://worldfloristassociation.com/ . The site is so big that the system
hangs each time. Please help

Rishi

Original bug listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1421981&group_id=137793
&atid=739383

Steps to Reproduce:
1) Setup a web server so that
http://www.mysite.com/sitemap.xml.gz 302 redirects to
http://www.anothersite.com/www.mysite.com/sitemap.xml.gz
(I'll call this the REDIRECT_LOCATION).
2) Create a config xml file with some URLs and the
config values: suppress_search_engine_notify="0",
base_url="http://www.mysite.com", verbose="1"
store_into="<webserver path to REDIRECT_LOCATION>"
3) Run the generator.

Expected Results / Output:
Generator runs, creates sitemap, and notifies Google
with no warnings/errors.
ie:
Notifying search engines.
Notifying: www.google.com

Actual Results / Output:
Generator runs, creates sitemap, and outputs errors
before notifying Google.
ie:
Notifying search engines.
[ERROR] When attempting to access our generated Sitemap
at the following URL:
http://www.mysite.com/sitemap.xml.gz
we failed to read it. Please verify the store_into
path you specified in
your configuration file is web-accessable. Consult
the FAQ for more
information.
[WARNING] Proceeding to notify with an unverifyable URL.
Notifying: www.google.com

Note on 302 Redirects:
Google Sitemaps honors 302 redirects to sitemaps, so
the Generator should too. This is used in our company
to host all our client-site's sitemaps on one webserver
instead having to move each individual sitemap onto the
client's server.

The original bug is listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1676398&group_id=137793
&atid=739383

Hello,

I try to generate sitemap with access logs method.
When the script parses the log file, I take the errors as shown below:

Opened ACCESSLOG file: /home/[mysite]/logs/[logfile.gz]
Traceback (most recent call last):
File "sitemap_gen.py", line 2203, in ?
sitemap.Generate()
File "sitemap_gen.py", line 1775, in Generate
input.ProduceURLs(self.ConsumeURL)
File "sitemap_gen.py", line 1115, in ProduceURLs
for line in file.readlines():
File "/usr/local/lib/python2.4/gzip.py", line 414, in readlines
line = self.readline()
File "/usr/local/lib/python2.4/gzip.py", line 388, in readline
c = self.read(readsize)
File "/usr/local/lib/python2.4/gzip.py", line 225, in read
self._read(readsize)
File "/usr/local/lib/python2.4/gzip.py", line 277, in _read
uncompress = self.decompress.decompress(buf)
MemoryError

Can anyone help me with this?

thanks,
Soulis

[deleted issue]

Original bug listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1498163&group_id=137793
&atid=739383

# ./sitemap_gen.py --config=forex-rdc.ru.xml
Reading configuration file: forex-rdc.ru.xml
Opened URLLIST file: forex-rdc.ru.txt
Walking DIRECTORY "/usr/local/www/forex-rdc.ru/images/"
Walking DIRECTORY "/usr/local/www/forex-rdc.ru/demobook/"
Walking DIRECTORY "/usr/local/www/forex-rdc.ru/"
Opened ACCESSLOG file:
/var/log/httpd/access_forex-rdc.ru.log
[WARNING] Discarded URL for not starting with the
base_url: http://admin/
[WARNING] Discarded URL for not starting with the
base_url: http://styles.css
[WARNING] Discarded URL for not starting with the
base_url: http://images/predictions_small.gif
[WARNING] Discarded URL for not starting with the
base_url: http://www.example.com
[WARNING] Empty URL
Sorting and normalizing collected URLs.
Writing Sitemap file
"/usr/local/www/forex-rdc.ru/sitemap.xml.gz" with 50000
URLs
Sorting and normalizing collected URLs.
Writing Sitemap file
"/usr/local/www/forex-rdc.ru/sitemap1.xml.gz" with
50000 URLs
Sorting and normalizing collected URLs.
Writing Sitemap file
"/usr/local/www/forex-rdc.ru/sitemap2.xml.gz" with
50000 URLs
Sorting and normalizing collected URLs.
Writing Sitemap file
"/usr/local/www/forex-rdc.ru/sitemap3.xml.gz" with
50000 URLs
Sorting and normalizing collected URLs.
Writing Sitemap file
"/usr/local/www/forex-rdc.ru/sitemap4.xml.gz" with
50000 URLs
Sorting and normalizing collected URLs.
Writing Sitemap file
"/usr/local/www/forex-rdc.ru/sitemap5.xml.gz" with
50000 URLs
Sorting and normalizing collected URLs.
Writing Sitemap file
"/usr/local/www/forex-rdc.ru/sitemap6.xml.gz" with
50000 URLs
Sorting and normalizing collected URLs.
Writing Sitemap file
"/usr/local/www/forex-rdc.ru/sitemap7.xml.gz" with
50000 URLs
Sorting and normalizing collected URLs.
Writing Sitemap file
"/usr/local/www/forex-rdc.ru/sitemap8.xml.gz" with
50000 URLs
Sorting and normalizing collected URLs.
Writing Sitemap file
"/usr/local/www/forex-rdc.ru/sitemap9.xml.gz" with
50000 URLs
Sorting and normalizing collected URLs.
Traceback (most recent call last):
File "./sitemap_gen.py", line 2203, in ?
sitemap.Generate()
File "./sitemap_gen.py", line 1775, in Generate
input.ProduceURLs(self.ConsumeURL)
File "./sitemap_gen.py", line 1137, in ProduceURLs
consumer(url, True)
File "./sitemap_gen.py", line 1843, in ConsumeURL
self.FlushSet()
File "./sitemap_gen.py", line 1862, in FlushSet
url.priority = '%.4f' % (float(dup) /
float(self._dup_max))
MemoryError



Contact me via e-mail: [email protected]

Original feature listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1236428&group_id=137793
&atid=739386

I'm running a purely PHP website, python isn't even in
the FreeBSD Ports Collection I'm using to maintain the
servers. Thus, I am unable to use Python on any
portion of the site.

Can someone make a PHP version of this Google feature?

See: https://bugs.launchpad.net/bugs/670186

There's a typo, and one branch of an if-else sets sitemap__index_header instead 
of sitemap_index_header.

[deleted issue]

Original feature listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1466690&group_id=137793
&atid=739386

Currently the <directory> tag in the config.xml has an
optional "default_file" attribute that specifies the
name of the index file. It would be nice to also allow
some sort of "default_file_required" attribute that
would exclude all the subdirectories that do not have a
default_file in them.

The source code mixes tabs and spaces for indentation. In Python, this is very 
dangerous, as Python treats all tabs as exactly equal to 8 spaces, unlike 
almost all editors.

See http://www.python.org/dev/peps/pep-0008/

Version of script: 1.5
OS: unix

The opening xml tag in the generated sitemap_index file is incorrect, and
causes the whole xml file to be malformed. The opening tag reads:
<urlset
  xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
  http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">

instead of 

<sitemapindex
  xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
  http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd">


Cause(s) of issue:

Line 1800 in sitemap_gen.py should read:
sitemap_index_header = NEWS_SITEINDEX_HEADER
instead of
sitemap_index_header = NEWS_SITEMAP_HEADER

Line 1802 in sitemap_gen.py should read:
sitemap_index_header = GENERAL_SITEINDEX_HEADER
instead of 
sitemap__index_header = GENERAL_SITEMAP_HEADER

Original bug listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1521152&group_id=137793
&atid=739383

Hello!
when i'm trying to add such option to my config.xml file
<sitemap path="I:\WWW\my_site\sitemap*.xml" />
error occurs, but unfirtunately i can't copy it here,
so i'm giving a link to print screen
please look
http://img132.imageshack.us/img132/3346/untitled20tt.jpg
Waiting for your response.

[deleted issue]

I see no recent activity except spam comments. Is there anybody still doing 
builds and maintaining the issues? Depending on the answer I want to either 
post a patch or find a different tool. Cheers, MetaEd

Original feature listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1220254&group_id=137793
&atid=739386

Can I suggest adding an additional filter to the basic
config.xml file to ignore files created using frontpage.
Just add :
<filter action="drop" type="wildcard" pattern="*_vti_*" />

It does work, and ignores repetition
Kevan Wilding
[email protected]

What steps will reproduce the problem?
1. sitemap type = web
2. more then 50.000 Links, so at least two sitemaps are created like : 
sitemap.xml, sitemap1.xlm
3. run the script 

expected is to write the sitemap_index.xml correctly, but the script fails.
Traceback Information : 

The Sitemap type is WEB Sitemap.
<type 'instance'>
Walking DIRECTORY "/rotek/inetpub/pics.rotek.at/"
Sorting and normalizing collected URLs.
Writing Sitemap file "/rotek/inetpub/pics.rotek.at/sitemap.xml" with 50000 URLs
Sorting and normalizing collected URLs.
Writing Sitemap file "/rotek/inetpub/pics.rotek.at/sitemap1.xml" with 12037 URLs
Writing index file "/rotek/inetpub/pics.rotek.at/sitemap_index.xml" with 2 
Sitemaps
Traceback (most recent call last):
  File "/rotek/inetpub/SiteMapCreator/sitemap_gen.py", line 2092, in <module>
    sitemap.Generate()
  File "/rotek/inetpub/SiteMapCreator/sitemap_gen.py", line 1665, in Generate
    self.WriteIndex()
  File "/rotek/inetpub/SiteMapCreator/sitemap_gen.py", line 1810, in WriteIndex
    fd.write(sitemap_index_header)
UnboundLocalError: local variable 'sitemap_index_header' referenced before 
assignment



the error is caused by a typo in the code, procedure WriteIndex, Line 1802 :

Original Code : sitemap__index_header = GENERAL_SITEMAP_HEADER
Correct Code :  sitemap_index_header = GENERAL_SITEMAP_HEADER

please find attached the corrected sitemap_gen.py

[deleted issue]

[deleted issue]

[deleted issue]

[deleted issue]

What steps will reproduce the problem?
1.  tar -xvzf sitemap_gen_1.5.tar.gz

What is the expected output? What do you see instead?
Expected the files to be in a directory sitemap_gen_1.5/ instead they are 
placed in the current directory, mixed with everything else.

What version of the product are you using? On what operating system?
1.5

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.

hi,your plugin is a good thing. but I want to generate a URL with chinese word 
.can it ?

it always like this ——http://www.kuen.me/%e5%85%b3%e4%ba%8e/



but this is what I want ----- http://www.kuen.me/关于


If you can solve my problem, be grateful

What steps will reproduce the problem?
1. changing the SSL port
2. typing https://localhost:8181/

What is the expected output? What do you see instead?
I can access Google Sitemap generator by typing
http://localhost:8181/cgi-bin/admin-console.cgi, but I cannot access it
remotely.
Once I try to enable https, I cannot access the admin console anymore, and
I received the following error "Data Transfer Interrupted". I cannot access
https://localhost:8181/in both Firefox and IE. Thanks for your help.  


What version of the product are you using? On what operating system?
I am using the most recent version, and the operating system is Windows XP


Please provide any additional information below.
I think that the main problem is how to enable https correctly.  Thanks.

What steps will reproduce the problem?
1. Using the third party tool generator
http://www.xml-sitemaps.com/standalone-google-sitemap-generator.html

The tool created the Sitemap with no issues for the following website

www.techwebsupport.com

However when entering the Sitemap to the Sitemap section on the web master
tools the status sits at "Error", Upon checking the error details it tells
me that the Sitemap is in the incorrect format, This tool created the
Sitemap in xml as Google states it should be and the tool is recommended by
Google.


I am currently using Vista 32bit however it should not cause any issues as
the tool is based online.

This is the fourth "Google Sitemap Generator" and its the fourth time I am
receiving the error.

Regards







Craig

Version of script: 1.5
OS: windows xp

I tried to generate a sitemap using the urllist method. 
When the script writes the sitemap_index file the following error occurs:

The Sitemap type is WEB Sitemap.
...
Writing index file "C:\tools\sitemapGenerator\sitemap\sitemap_index.xml"
with 2 Sitemaps
Traceback (most recent call last):
  File "sitemap_gen.py", line 2092, in ?
    sitemap.Generate()
  File "sitemap_gen.py", line 1665, in Generate
    self.WriteIndex()
  File "sitemap_gen.py", line 1810, in WriteIndex
    fd.write(sitemap_index_header)
UnboundLocalError: local variable 'sitemap_index_header' referenced before
assignment

There's an extra underscore when referencing the sitemap_index_header
variable in the WriteIndex method, line 1802. The line reads

sitemap__index_header = GENERAL_SITEMAP_HEADER
versus
sitemap_index_header = GENERAL_SITEMAP_HEADER


Thanks,
Leah

i m working in perl.
How can implement "exclude directory" for sitemap generator.
i can' able to implement exclude directory in sitemap.

untar litters the users pwd with files. instead it should create a directory 
that contains the files. 

petrichor:sitemaps$ tar zxvf sitemap_gen_1.5.tar.gz 
./AUTHORS
./COPYING
./ChangeLog
./PKG-INFO
./README
./._example_config.xml
./example_config.xml
./._example_urllist.txt
./example_urllist.txt
./setup.py
./sitemap_gen.py

[deleted issue]

[deleted issue]

[deleted issue]

I've tried to install this tool using the setup.py but it doesnt seems to work. 
pip also have problems with it.

Is this normal ? I can have a look to the setup.py file and fix that if you 
update then the records on pypi (basically via a "python setup.py sdist upload" 
once my fixes commited).

I also need access to the SVN to do so.
Thanks,
Alexis

[deleted issue]

[deleted issue]

[deleted issue]

The example code shows the variable name as "supress_search_engine_notify" 
but the reference guide shows it as "suppress_search_engine_notify". 

Original feature listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1265664&group_id=137793
&atid=739386

Ability to specify files and have links crawled from
there. This is important for dynamic sites with rewrites.
The links are to static files, but they technically don't
exist on the server for programs to crawl.

[deleted issue]

Original bug listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1526064&group_id=137793
&atid=739383

Include Filters Applicable to URLs.      Private: (?)
No
I would like to apply a filter, for example a regular
expression, to autmatically change the URLs, for exmaple

<filter action="modify" type="regexp"
pattern="s/\.html.en$//" />

where s/.../../ is a regexp substitution.

This might be very usefull if you use Apache's
MultiViews or Type Maps and if you want to sanitze your
URLs.

Original feature listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1223364&group_id=137793
&atid=739386

The recommendations for Google are just to submit html
pages (as opposed to .gifs, .jpgs, etc) So either give an
example in the config if there is a simple way to reject
ALL else using say "regexp" for those not familiar with
regexp, or make a new switch to make it easier to pass
JUST .htm/.html. Right now we had to list and filter all
other possible extensions using the wildcard filters since
it was not acceptable to just pass ALL .htms since
there were some .htm's calls found in the logs with
parameters which we did NOT want to include.

Also it would be nice to have an option for the number of
levels walked in directories. For instance we wanted to
have our root, and only a PORTION of the subdirectories
contained in it walked. Since walking the root apparently
automatically walks ALL subdirectories, the only way
we could think to do this was to filter the rest out by
name. Would be nice to be able to specify if walking the
root included walking ALL subdirectories or only
specified ones.

What steps will reproduce the problem?
1. Using Debian 6.0.1, which runs Python 2.6.6 by default I get a deprecation 
warning with module md5.

What is the expected output? What do you see instead?

No warning should be issued.

What version of the product are you using? On what operating system?

1.5 under Debian 6.0.1 "Squeeze".

Please provide any additional information below.

I'm attaching a trivial patch which fixes the problem.

Original feature listed here:
http://sourceforge.net/tracker/index.php?func=detail&aid=1265634&group_id=137793
&atid=739386

It would be great to be able to specify that for all files
with a certain file extension (e.g., *.php), that when
determining the last modified date, you could have it
take the maximum date of the file itself and all include
files - specified either as an extension (e.g., "*.inc.php"),
or as a specific list of include files.

Right now, if we modify an include file, none of the main
files on the site actually change (in terms of the modify
dates.)

So, for example, be able to specify something like:

<include forExtension="*.php"
includeExtension="*.inc.php" />

or

<include forExtension="*.php">
<file name="example.inc.php" />
<file name="includes/example2.inc.php" />
</include>

small fix for sitemap_index generation