When running with "Fix and Enhance Data" option, I get an "Error converting implement113 Blue Orb of Dragonkind" message.

Log:

10:06:54 [INFO] Exporting category Poison in thread pool-2-thread-7.
10:06:55 [WARNING] Export failed: java.lang.UnsupportedOperationException: Error converting implement113 Blue Orb of Dragonkind
	at db4e.converter.Convert.convert(Convert.java:419)
	at db4e.controller.Controller.lambda$exportEachCategory$27(Controller.java:675)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 5
	at db4e.data.Entry.getSimpleField(Entry.java:60)
	at db4e.converter.Converter.meta(Converter.java:436)
	at db4e.converter.ItemConverter.correctEntry(ItemConverter.java:377)
	at db4e.converter.Convert.convertEntry(Convert.java:457)
	at db4e.converter.Converter.convertEntry(Converter.java:47)
	at db4e.converter.ItemConverter.convertEntry(ItemConverter.java:61)
	at db4e.converter.Convert.convert(Convert.java:408)
	... 4 more

10:06:55 [INFO] Status: Error converting implement113 Blue Orb of Dragonkind.

Exporting with "Fix and Enhance Data" disabled does not produce this error message.

Hello, I adore the project you've done. My 4e group has been running a Living campaign for the past 6ish years, and we've got a fairly extensive list of compendium errors we've located and corrected for our offline version. I figured I would share them with you in case you wanted to implement them here
https://docs.google.com/document/d/1yhKzsuMKoWImvvWaVRXJatlrYNvA_nWwKBRso93qAks/edit

• Divine domain lookup.
• Power keyword chained by semi-colon and or (e.g. power7432 Master's Surge)
• Keyword in "Keywords" section (e.g. power13053 Give Chase)
• Exclude theme / pp / ed descriptions from index. Push to 3.7 with new options
• Power with wrong frequency colour (and thus wrong deduced meta), e.g. power9964 Boughs of the World Tree
• Check that power fixes are also applied to embedded form, e.g. said power in epicdestiny510 World Tree Guardian
• Find and fix flavor with wrong HTML e.g. eyes of selune.
• Teleport power without teleportation keyword (e.g. power12454 Shadow Adept)
• Spirit power without spirit keyword (e.g. power12880 Cloudburst)
• Fix glossaries with lower case titles.
• Sort glossaries by name.
• Add ability descriptions and lookup.
• Fill in implement rules (RC p.274).
• Fill in bloodline rules.
• Add glossary type to title.
• Check that item fixes are also applied to item set. (like power above) Push to 3.7
• Reclassify glossary rules, e.g. Character, Exploration, etc. Push to 3.7

I've build the latest revision of the project and opened it in Internet Explorer 11.

The Refresh Categories and Check works fine, however, Get All Items and Get new + changed cause a popup that explains a login window will appear etc.. However, no window appears.

I do have popup blocker turned off so that doesn't appear to be the issue. Should this work fine with IE 11?

I had hopped that ObservableList is synced. Now I know it isn't.

The program in the browser is written all in Chinese, how do i change to english?

Steps to reproduce the error

Download 4e_compendium_downloader.jar and run it using
java -jar 4e_compendium_downloader.jar

Expected result

Application starts

Actual result

$ java -jar 4e_compendium_downloader.jar 
Error: Could not find or load main class db4e.Main

System information

OS: Arch Linux

$ uname -a
Linux ra 4.8.4-1-ARCH #1 SMP PREEMPT Sat Oct 22 18:26:57 CEST 2016 x86_64 GNU/Linux

Java version: java-8-openjdk

$ java -version
openjdk version "1.8.0_112"
OpenJDK Runtime Environment (build 1.8.0_112-b15)
OpenJDK 64-Bit Server VM (build 25.112-b15, mixed mode)

• Going from view to list does not remember scroll pos.
• Cannot search "0 gp".
• "0 gp" should be sorted below empty.
• Support keyboard for next/prev.
• Fix "Browse" title.
• Add a way to go to category from entry, may be a hover link.
• Display perma link on Chrome.
• Go from entry to blank list should auto redirect to the entry's category.
• Check cloth armour lookup.
• Fix column search box focus lost issue in Edge.

Current:

1. Language and symbol set can be changed in about page.
2. Show/hide highlight can be changed in Firefox context menu.
3. Language is remembered; symbol set and highligh state is not.

Fix:

1. Show all settings in both about and context menu.
2. Make sure all settings is saved in localStorage.

Fix screen resize not triggering mobile state switch.

Viewer's current mobile layout was not developed with changing screenwidth in mind. But it is now very common with window snapping and pad/convertibles that often switch between vertical and horizontal.

Ran the downloader no problem. Had the "Fix and enhance data" selected and tried exporting and received the error "Error converting implement113 Blue Orb of Dragonkind." Tried a couple of more times with the same result. Unchecked "Fix and enhance option" and successfully exported. Tried again to export fixed data and again received error. Copied log starting from successful export and ending with most recent unsuccessful export.

Thanks for making the tool! Even if I don't have the fixed data, just being able to access something offline is brilliant!

compendium downloader.txt

Version 3.5.1 (Released)

Style

• Go through each category and fix entry style.
• Mobile friendly listing layout.

Functionality

• List paging with a fixed size of 200 to 1000 rows.
• Preserve filter when changing only search term.

Version 3.5.2 (Released)

Functionality

• Internal entry linkage.

Internal

• Shorten url by dropping category part.
• Consolidate data file to reduce file count.
• Multi-thread indexing and export.
• Merge convert with export to balance disk speed with cpu speed.

Version 3.5.3 (Released)

Functionality

• Use datalist to support autocomplete in search fields.
• Finish remaining UI refinements.
• Add race origins and use as Type in Everything.
• Finish real level and cost filter for items.
• Add csv, tsv, excel, and sql export.
• Update viewer help (typo, grammar, links etc.).
• Highlight column with column filters.
• Record a video walkthrough.

Internal

• Refactor converter code to make it cleaner and better safeguard regx match.

Version 3.5.4 3.6

Functionality

• Add option to not apply fixes for export.
• Allow export of fixed data.
• Can we add web app support?
• Explore the possibility of apk export.
  Not feasible; need to bundle apk tools to insert assets. And the UI does not behave like Android.
• Allow to jump directly to item by searching id.
• Support swipe.

Internal

• Compress jsonp data to further reduce file size and number.
• Add conversion test cases.
• Load initial data in background in downloader.
• Split more viewer resources out to res folder to reduce size and speed up loading.

As title

• Re-classify Lair items.
• Fix artifact source books and lack of warnings.
• Incorrect armour type conversion of Armor: All (e.g. Runic Armour)
• Update mundane shields' types column to match magical shields.
• Fix Scorned Noble's benefit column.
• Change class's ability column with primary ability.
• Add Artifact and Item Set rules.
• Add size and keywords for monsters and companions.

v3.0 aim: On par with online compendium

Milestone 1 (released)

Style

• Implement content style.
• Rewrite content image to unicode symbols.
• Displayed symbol set demo and selection. (Previously "Add test for replacement symbols and advise on font.")

Documentation

• Write documentation on data reterival.
• Copy documentation on data search.

Milestone 2 (released)

UI

• Make download page displayed status consistent with actual status (currently refresh cat will overwrite).
• Allow re-index extended columns of existing data.
• Allow delete local category.
• Show search result count in category.
• Add "Search Name", "Search Content", "Search Advanced" buttons and disable auto search.
• Fix title bar at top.

Style

• Beautify background and buttons.

Milestone 3 (not released, continue in v3.5)

UI

• Start search only after both search method and category are selected.
• Add buttons to navigate up/down search result.
• Implement dynamic url.
• Apply live language change to title and category names.
  
  Use swipe left / right to navigate up / down search result. (Incompatible with non-fixed viewport)

Functionality

• Add by-field search.
• Add sorting.
• Add search terms highlight.
• Hide sort columns.
• Update to works with latest compendium and IE 11.

Alpha

UI

• Add/Adjust animation, gradiant, shadow, font, and other effects.
• Improves responsive UI.
• Add a "loading" screen for loading data, and detect slow locations (e.g. online or extSDCard).

Functionality

• Handle various network error, malform error, missing file errors, without ugly alerts.
• For non-english localisation, add link to google translate.
• Refactor and clean up code.
  
  Delete local entries that doesn't exist on remote. (With discontinuation of updates, no longer necessary)

Beta

Functionality

• Fix any outstanding bugs.

Deployment

• Test on IE 9/10/11, Firefox, Chrome, desktop Safari
• Test on Android browser / Chrom / Opera / Firefox, Mobile Safari.

These issues should be fixed before release.

• Fix icon background on help page in all browsers.
• Detect load error in Chrome for Android cannot reproduce error
• Test Java 9
• Test Opera mobile
• Update no js message for Android HTML Viewer
• Correct Item434 fix.

Hi,

I've used your program on my desktop PC no issues (awesome program!), but when I try to use it on my Android tablet I recieve the "Loading Java Script" message in Chrome.

I have tried clearing the cashe, downloading Firefox, re-enabling Java, and still nothing brings me past that page. I have made sure that both the .html shortcut and 4e_database_files folder are in my internal storage, as well.

Is a fix for this? I'm not a very tech-savvy person, but if there are simple steps I can likely follow them.

Thanks!

Tablet: Samsung Galaxy E
Running: Android 7.1.1.
Chrome Version: 65.0.3325
Firefox Version: 59.0.3 (Java enabled)

Some data lists, such as Power, Monster, Item, Feat, is very long and takes a while to download.

Would be nice to see whether it is possible to show download progress so that status can be updated.

• Only load the index that is required.
• Basic armors are not linked.
• Chainmail, Bardings, and Shields need alias.
• Identify class features and link to the class.

I've been trying to use the 4e_database.html
On the update page, after checking and hitting the get all button, I sign in. A new pop up window appears that displays one compendium entry. For the Class, it opens Cleric (Templar). For Disease, it opens Wererat Filth Fever.

Title. The database site can no longer be accessed. Even from a cached version of the website, it couldn't read information from the database.

In addition, Digital River's storefront for DDI subscriptions appears to have shut down, so there is no longer a way to purchase DDI subscriptions to gain offline access to the compendium.

I've been really enjoying using the database for my current 4e game, and my players have been using it for their characters as well, so I'd like to know what steps I should follow from here.

Release of ver 3.0

• Empty search with sort is parsed incorrectly.
• Exclude ritual flavor text from text index.
• Exclude paragon path flavor text from text index.
• Exclude epic destiny flavor text from text index.
• Detect and hopefully fix flavor text that contains html.
• Powers should be sorted by Class, Level, Frequency, then Name
• Hybrid classes should be sorted last.
• Move "Artifact" text from Type column to Rarity column.
• Update frontpage video placeholder.
• Rename "app" to "program" for the downloader.
• Fix cat link layout issue e.g. Arrow (weapon219)
• Don't jump to entry view on drag so that it is possible to select text on desktop.
• Fix item name selection results in rarity lookup.
• Fix prev/next button covered by content (mobile only?)
• Fix long power class overlaping long power name (e.g. World Serpent paragon)
• glossary670 creates a blank lookup entry.
• Fix non-ascii lookup issue (e.g. Eyes of Selûne)
• Check why lookup "Dragon" give "Drag".
• Searching "AV2" in sourcebook column of glossary yields non-AV2 results.
• Switch language default to English.
• Add quick lookup to context menu.
• Toggle highlight menu label when on/off.
• Remove viewer search cache to simplify code and save memory.
  Aborted because it speeds up going back to list
• Allow control of thread count.
• Catch that dreaded OutOfMemoryError.

Many download and export messages are using present tense for sensible error message, like "Get power data" may becomes "Get power data failed".

But it really should says "Getting power data" instead, so that users won't mistake it as a command.

Target 3.5.2.

• Fix special characters in name.
• Fix monk powers names.
• Vecna's Boon of Diabolical Choice (check price)
• Mark of the Star (and friends) (check price)
• Shivli, White Wyrmling (check price)
• Sun's Sliver level
• Anarusi Codex cost and format
• Spot the Path double colon. Anything missing?
• Vampire class healing surge.
• Essential hybrids are from dragon 400 not 402.
• Don't text-index individual items in item set.
• Add skill name to skill power class.
• Add power type (Attack/Utility/Feature/Feat) to meta.

May never happens. Who knows.

Viewer Features

• Display all search / filtered result on one page.
• Right click remove entry from multi-entry display.
• Right click add / remove entry to "my favorite".
• Display said "my favorite".
• "Character Builder" mode to filter theme/power/feat/PP/ED by race and class.

Viewer Structure

• New layout. Use flex and keep things simple. Better loading indicator.
• Use SharedWebWorker to share data across multiple viewers.
• Failing that, use WebWorker to preload and decompress data in the background.
• Convert to ES6 and rewrite with Promise.

When category is switched, e.g. from All to Item, the result count would be lost on all other categories.

When using Windows 10's High Contrast (Black) and Firefox, it becomes white text on yellow or light grey background, impossible to read.

Edge should have same issue but cannot load. Chrome is normal.

Whenever I try to connect I get this error. Login and password are fine since I cut and pasted the same one directly to the DDI website and it works fine. Tested under both Windoes10 and Mac High Sierra. I also have a subscription.

Thanks.

Improvements and advanced functions that didn't exist in previous generations

Functionality

• Multiple result mode.
• Add parsing of extended columns, such as item levels or power keywords.
• Build name index and create links from content.
• App cache update, so that the data can be sync-ed.

Problem: Cannot select an item in the list unless a catagory button is selected.
Desire: Generally the search result provides the desired item within the first few rows of the result list (Good job on the search routine).
Fix: The problem is in the - funciton act_list_row_click() - the 'cat' object is not set correctly when 'All Catagory' was last searched.
Change line 3773:

if ( ! cat ) cat = this.getAttribute( 'data-category' );    //original

to

if ( ! cat ) cat = od.data.get( this.getAttribute( 'data-category' ));      // FIXED

• Add reroll rules, may be proficiencies, and check existence of other common rules.

Hi,

I installed Java 8 Update 111 (October 18, 2016) and ran 4e_compendium_downloader.exe

I am getting the error: Not on FX application thread; currentThread = pool-3-thread-1
Here is the debug log:

18:53:56 [FINE] Testing login finished normally.
18:53:56 [INFO] Status: Getting race template.
18:53:56 [FINE] Getting race template finished normally.
18:53:58 [INFO] Status: Getting race data.
18:53:58 [FINE] Getting race data finished normally.
18:53:59 [INFO] Status: Parsing race.
18:54:00 [INFO] Status: Listing race.
18:54:00 [FINE] Listing race finished normally.
18:54:00 [WARNING] Download failed: java.lang.IllegalStateException: Not on FX application thread; currentThread = pool-3-thread-1
at com.sun.javafx.tk.Toolkit.checkFxUserThread(Toolkit.java:236)
at com.sun.javafx.tk.quantum.QuantumToolkit.checkFxUserThread(QuantumToolkit.java:423)
at com.sun.javafx.webkit.prism.PrismInvoker.checkEventThread(PrismInvoker.java:62)
at com.sun.webkit.dom.JSObject.getMember(JSObject.java:61)
at db4e.controller.Crawler.toArray(Crawler.java:151)
at db4e.controller.Crawler.openCategory(Crawler.java:105)
at db4e.controller.Controller.downloadCategory(Controller.java:362)
at db4e.controller.Controller.lambda$startDownload$9(Controller.java:334)
at db4e.controller.Controller.lambda$runTask$21(Controller.java:496)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

18:54:00 [FINE] State: Can Download
18:54:00 [INFO] Status: Not on FX application thread; currentThread = pool-3-thread-1.
18:54:00 [FINE] Progress: 0.
18:54:09 [FINE] State: Can Export
18:54:09 [INFO] Status: null.

Thanks!

Version 3.6.2

Viewer

• Add search to text selection's context menu. Done in 3.6
• Fix monk power [Attack Technique] showing up in entry title.
• Expand sparrow event attachment to support passive events.
• Linkify entry and page title so they can be ctrl-clicked to clone in new tab.

Downloader

• Add feat type (multiclass, bloodline, style, dragonmark)
• Add range type to power keywords.
• Add ritual category.
• Add reroll rules, may be proficiencies, and check existence of other common rules. (Reroll rules not found; proficiencies are spread around in glossaries)
• Add mundane weapon keywords to type.
• Add user agent string override.
• Add mid-category progress feedback.

Version 3.7

Viewer

• Fix ritual and monster layout on mobile (shift two col to one)
• Toggle multi-column display for multi-entries. (Single entry remains centered)
• Enable show all / show page button.
• Fix duplicate load request of multiple results and enable the function.
• Sorting results and remove results from display, with url update.
• Preserve filter and list in url (?)
• Preview matching sentence(s).
• Options to show racial / skill / associated powers in racial / skill / feat entry.
• Option to hide flavour text.
• Collapse / Expand embedded powers.
• A button to register as a windows program?

Downloader

• Replace unicode symbol with icon. (with proper alt text).
• Split flavour text index from text index.
• Add "Power Keywords" to HC, PP, and ED.
• Add "Abilities" to PP, and ED.
• Optimise normalise() which takes the most time.

1. Fix grammar mistakes.
2. Fix image links.
3. Add links to resources and examples.

Currently, after data is saved, the 'saved' count does not update.

Search often yield long result. Would be nice to have a 'top' button.

Perhaps it would be nice to replace update button with top when the page is not already at top.

As title.

Do a name search for "back" in All Category. All backgrounds entry match the result because they belongs to the "background" category. They shouldn't.

Sorry for the confusion on my part, but I've run the downloader and have downloaded the database information, but I don't understand what I'm suppose to do next? I've downloaded trpg-dnd-4e-db, but all I see are xml files that I can't excute. I also have CocoDoc, but it also only has xml files that I can't run. I'm a complete noob when it comes to this stuff, so any more information you could give me would be greatly appreciated.

Also, is it possible to export as a CSV file? I've tried to download SQLJet to test a conversion process, but the download keeps freezing.

We can try to use less memory by not keeping item contents.

This can be done for browsing action, but can also be done for update action. If update is simplified to save at real time, that part should become simpler, not only to maintain but also to use.

Things too late to do in 3.5.1:

• Split items off into weapon, implement, and armor/shield.
• Split and group Artifacts.
• Regroup superior implements.
• Regroup consumable wonderous items.
• Group Item Sets.
• Group weapon, implements, armor.
• Set level for base weapons, implements, armors.
• Group alternative rewards, standards, tattoos, shards, mounts, figurines.
• Fix Kord's Relentless, Xenda-Dran tier, Orium Implement type, Shield of Prator style, Primal Grove cost.
• Fix Rings of the Akarot.
• Moves Assassin Poison to poison section and fix style.
• Fix items that say "reproduced below" but doesn't.

This requires rewriting the current index process to better separate downloaded entries and exported entries.

The data entity glossary679, the "Familiar monster keyword" with an empty body from Dungeon Magazine 211, is invalid and should not exist in the first place.

Now we need to find a way to not export it without triggering any consistency checks in multiple layers.

• A system to check that entry fixes are also applied in embedded form.
• Reclassify glossary rules, e.g. Character, Exploration, etc.
• Add "Varies" keyword (e.g. power5888, maybe needed for power13421 and others).
• Add Hazard type (Blaster, Lurker etc.) to Group.
• Make single-use terrain layout consistent and fix trap943,trap947,trap948

Relying on ActiveX and JavaScript to download is causing lots of trouble.
Since ActiveX is discontinued in Egde, and that JavaFX has good built-in support, now is a good time to switch to Java.
It will be less user friendly, but more powerful, easier to maintain, and cross platform.