Giter Club home page Giter Club logo

mtgjson3's Introduction

MTGJSON Logo
MTGJSON Version MTGJSON Version Date

MTGJSON is an open-source repository of Magic: The Gathering card data, specifically in JSON and CSV formats. This repository contains the build scripts we use to create our data sets.

How to Contribute

Running a project as large as ours isn't easy, and we rely on the community to help keep our project going. The community can support us through two different ways: financial contributions and code contributions.

Github Sponsors
We accept reoccurring donations via GitHub Sponsors, that grant priority support from MTGJSON maintainers and a special role on our Discord indicating your support. This is the preferred approach for financial contributions, as the MTGJSON team pays no fees for these types of donations!

Patreon
We accept reoccurring donations via Patreon, that grant priority support from MTGJSON maintainers and a special role on our Discord indicating your support.

PayPal
We accept one-time donations via PayPal, for those who want to say thank you to the project.

Code Contributions
We love it when the community contributes back to the project! If you'd like to help improve our data for the hundreds of projects and stores we support, do reach out via Discord!

Connect With Us

Discord
The team stays in contact via Discord. The server is open to the public and is a great place to collaborate with other like-minded people. Stop by today and say hi!

About Us

The Team

The MTGJSON team has been led by Zach Halpern since 2018, with support from an awesome group of people. The full team lineup can be found MTGJSON's homepage.

Our Product

MTGJSON at its core is a database that can be downloaded for offline access to Magic: the Gathering card data. We pride ourselves on our documentation, and aim for full transparency with the community.

Our Partners

Over time, MTGJSON has gone through a number of transitions to bring the best product for our consumers. We'd like to thank the following groups, in alphabetical order, for helping to support our mission by enriching our data:

How to Use

For 99% of our Users

MTGJSON supplies precompiled databases at https://mtgjson.com/api/v5/. This is the recommended way to use our service.

As stated before, we pride ourselves on our documentation which can be found at https://mtgjson.com/. If you find anything to be unclear or ambiguous, please open a ticket on our documentation repository so we can address your concern immediately.

We fully rebuild our API data once a week (on Monday afternoons) and our price dataset once a day. You can poll the Meta.json file to see when our data was last updated.

For those who want to build MTGJSON locally

Most of our users shouldn't have a need to build MTGJSON locally. However, there are always exceptions and we aren't ones to judge.

Build Box

While MTGJSON will work on Windows, Mac, and Linux, we prefer working within the Linux environment for lower overheads and less manual dependency management.

  • For Linux based build boxes (we recommend Ubuntu 20.04), your build box should have at least 2 cores and 4 GiB of RAM available.
  • For Mac based build boxes, your build box should have at least 2 cores and 4 GiB of RAM available.
  • For Windows based build boxes, your build box should have at least 4 cores and 8 GiB of RAM available.

Install Python3

MTGJSON is built on and tested against a wide range of Python3 versions. Currently, we maintain support for the following versions:

  • Python 3.8
  • Python 3.9
  • Python 3.10
  • Python 3.11
  • Python 3.12

Install MTGJSON

Local Installation

python3 -m pip install /path/to/mtgjson5/

PyPi Repository

We intend to put MTGJSON5 on the pip package archive in the near future, once the first set of revisions takes place.

Using MTGJSON

A fully up-to-date help menu can be achieved via python3 -m mtgjson5 -h, but for your convenience here is a recent rundown:

usage: mtgjson5 [-h] [-s [SET [SET ...]] | -a] [-c] [-x] [-z] [-p]
                [-SS [SET [SET ...]]] [-PB] [-R] [-NA]

optional arguments:
  -h, --help            show this help message and exit
  -s [SET [SET ...]], --sets [SET [SET ...]]
                        Set(s) to build, using Scryfall set code notation.
                        Non-existent sets silently ignored.
  -a, --all-sets        Build all possible sets, overriding the --sets option.
  -c, --full-build      Build new prices, MTGSQLive, and compiled outputs like
                        AllPrintings.
  -x, --resume-build    While determining what sets to build, ignore
                        individual set files found in the output directory.
  -z, --compress        Compress the output folder's contents for
                        distribution.
  -p, --pretty          When dumping JSON files, prettify the contents instead
                        of minifying them.
  -SS [SET [SET ...]], --skip-sets [SET [SET ...]]
                        Purposely exclude sets from the build that may have
                        been set using --sets or --all-sets.

mtgjson maintainer arguments:
  -PB, --price-build    Build updated pricing data then exit.
  -R, --referrals       Create and maintain a referral map for referral
                        linkages.
  -NA, --no-alerts      Prevent push notifications from sending when property
                        keys are defined.

MTGJSON Environment Variables

Due to how the new system is built, a few advanced values can be set by the user in the shell environment.

  • MTGJSON5_DEBUG When set to 1 or true, additional logging will be dumped to the output files
  • MTGJSON5_OUTPUT_PATH When set, MTGJSON will dump all outputs to a specific directory
    • Ex: MTGJSON5_OUTPUT_PATH=~/Desktop will dump database files to /home/USER/Desktop/mtgjson_build_5XXX and log files to /home/USER/Desktop/logs

Licensing

MTGJSON is a freely available product under the MIT License, allowing our users to enjoy Magic: the Gathering data free of charge, in perpetuity.

mtgjson3's People

Contributors

ancestral avatar bakert avatar bra1n avatar danielheath avatar dev-id avatar evie404 avatar fenhl avatar florentdouine avatar garbee avatar gwax avatar jceddy avatar kavu avatar kitroed avatar kmontag42 avatar leematos avatar lsmoura avatar maxspencer avatar mdurao avatar mjwybrow avatar naokiy avatar palladinium avatar petebeegle avatar pfirpfel avatar sembiance avatar silasary avatar starryknight64 avatar teh-maxh avatar tooomm avatar ynerant avatar zeldazach avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mtgjson3's Issues

Fated Reforged

Just letting you know Gatherer is updating as i type this message and that it's time for an update. Thanks for your work dude :)

Update: It's live on Wizards' Gatherer now 👍 !

Mishra's Toy Factory

Can you update pHHO to include Mishra's Toy Factory when you get a chance? :)

Thanks for your hard work!

imageName with numbers appended to card name.

In a good number of cards, the json imageName field is a string that ends with either a 2, 3, or 4. In these cases, without fail, the image is not found at mtgimage.com/card/[card name].jpg but is instead at mtgimage.com/card/[card name with number omitted].jpg.

I recognize that this might, instead be an issue with mtgimage, but as I cannot be sure, I opted for posting the issue here, as it appears to have more traffic than mtgimage.

Artist Corrections

Submitted corrections from a user:

  1. Certain printings of "Armored Griffin", "Just Fate", "Piracy", and "Town Sentry" have Brad Williams, though he goes by Bradley Williams on most cards and most printings.
  2. Certain printings of "Hamletback Goliath" have Brian Snoddy, without the accent found on all other instances.
  3. Certain printings of "Plains" have Christoper Rush, without an 'h'.
  4. "Keeper of Tresserhorn" has D.A. Gregory, instead of D. Alexander Gregory found on all other instances.
  5. "Curse of Thirst" promo printing has Dave Rendall, though it's actually Dave Kendall.
  6. "Rakdos Cackler" promo printing has Daven Kendall, though it's still just Dave Kendall. =]
  7. Certain printings of "Brainstorm" have DiTerrlizzi, with an extra 'r'.
  8. "Urborg Mindsucker" has both Tony Diterlizzi and Tony DiTerlizzi.
  9. "Counterspell" has Dom!, with an exclamation point, which matches the printed card but not the name on his other three cards.
  10. Certain printings of basic lands have Edward P. Beard Jr., without the comma found on his other cards.
  11. Certain printings have G. Darrow, instead of Geofrey Darrow.
  12. Certain printings of "Worn Powerstone" have Henry G. Higginbotham, instead of Henry G. Higgenbotham.
  13. Certain printings of "Mycoid Shepherd" have Jarreau Wimberley, instead of Jarreau Wimberly.
  14. "Fireball" and "Incinerate" have Jock, while "Cinder Cloud", "Dwarven Miner", and "Kukemssa Pirates" have JOCK in all caps.
  15. Certain printings of "Man-o'-War" have John J. Muth, instead of Jon J. Muth.
  16. "Mortal Wound", and "Ovinomancer" have Kevin Walker, though most cards have Kev Walker.
  17. "Boggart Ram-Gang" promo printing has Lucio Patrillo, instead of Lucio Parrillo.
  18. "Dark Hatchling", "Mana Leech", and "Spined Fluke" have Mark Nelson, instead of Mark A. Nelson.
  19. "Mana Crypt" has Matt Steward, instead of Matt Stewart.
  20. Certain printings have Matthew Wilson, instead of Matthew D. Wilson.
  21. Certain printings of "Grave Titan" have Nils Hamm (Lucas Graciano), though it's painted by Lucas Graciano and misprinted as Nils Hamm.
  22. Certain printings have Parente, instead of Paolo Parente.
  23. Certain printings have Randy Asplund, instead of Randy Asplund-Faith.
  24. "Swamp", and "Untamed Wilds" have Romas, instead of Romas Kukalis.
  25. Certain printings have Scott M. Fisher, instead of Scott M. Fischer.
  26. "Gitaxian Probe" promo printing has Trevon Claxton, instead of Trevor Claxton.

Add full foreign language support

Add full foreign language scrapes.

Implementation info:
Need to use ?printed=true on Gatherer to get this data.
If the cards have unique multiverseids probably have totally seperate json. Files like AllCards-x-ch.json or what not for each language.
When doing this, keep the 'foreignNames' field, and make sure it includes 'English' in non-english printings

Cache Issues

This is a great service. I was actually in the middle of scraping the data myself when I came across this. There is a fairly large problem with the caching implementation though, and that is that it never updates the cache. There should be some rule for how long a cache file stays valid, otherwise you'll never pick up on ruling updates. I believe this is what is causing your Theros file to be missing a LOT of rulings. Some are as far back as 9-15-2013 like the Whip of Erebos. I would be happy to help out with this, but I really don't want to dig into the java side of the project!

Split Vanguard (VAN) into two sets

Vanguard (VAN) consists of 32 physical cards and then a bunch of avatar cards that were online only. These should be split into two separate sets, making sure the online only one has an 'onlineOnly' set to true

Add MTGO only sets

There are several missing MTGO only sets that could be added:

  • PRM set. (promo cards)
  • Mirrodin Pure vs. New Phyrexia

Theme Decks:

  • Boltslinger
  • Exiler
  • Deathdancer Xira
  • Enchantress Rubinia

More sets?

I have an e-mail from someone that lists out a lot of the PRM cards, but I would need to find a more automated way of updating this list, probably looking into the MTGO game files directly to see if I can parse this info out or not.

Add tokens

Add Tokens. One place to get them is magiccards.info
Token List: http://magiccards.info/extras.html

Sadly magiccards.info doesn't have all of them and they are very slow to add new ones.

More token info:
https://github.com/Psithief/Cockatrice-tokens/blob/master/tokens.
http://mtgsalvation.gamepedia.com/Token/Full_List

Should do Issue #46 before this one, as the new tokens will go into the new tokens array.

Note: Afterwards, in addition to updating test_urls.js I also need to look at the tokens folder in DevLab/images/originals/mtg/CCGHQ/full/

foreign multiverse id

it might be useful to include the multiverse id in the foreign names field. this would allow me to fetch images in different languages

Add additional planes/schemes/vanguards

yawgatog.com has a few planes/schemes/vanguards that are not present on mtgjson
planes -- Celestine Reef, Horizon Boughts, Mirrored Depths, Tember City
schemes -- Drench the Soil in Their Blood, Imprison This Insolent Wretch, Perhaps You've Met My Cohort, Plots That Span Centuries, Your Inescapable Doom
vanguards -- Jace Beleren Avatar, The Avenger Avatar, The Harvester Avatar, The Hunter Avatar, The Philosopher Avatar, The Protector Avatar, The Warrior Avatar

Additional info:
Missing oversize cards. the planes and schemes are all promos. the planes are listed as the bottom 4 cards on -- http://magiccards.info/query?q=celestine reef&v=card&s=cname -- and the schemes
are listed as the bottom 5 cards on -- http://magiccards.info/query?q=plots that span centuries&v=card&s=cname -- the missing vanguards however i could not find anywhere on gatherer or magiccards.info. they appear to also be promos, but only available online.

Lastly, I should also look into additional data yawgatog may have that would be useful

Add 'color identity'

A user requested the following: "Commander/EDH, and color identity is what determines what cards can go in your decks for Commander. It refers to any colored mana symbol on the card, be it in the cost or text box. So, Yasova is URG, Talisman of Dominance is UB, Karn, Silver Golem is 'Colorless', the Pacts and Evermind are their respective colors as indicated by their Color Indicator, etc. Most cards are pretty straightforward, and just the color in the mana cost, but everything from Quagmire Druid to Memnarh to Lands (Sandsteppe Citadel) to Rattleclaw Mystic have color identities that differ than their casting cost.

http://mtgcommander.net/rules.php has an explanation of color identity (the official source of commander info/rules); it's also addressed in the comprehensive rules at 903.4 (http://magiccards.info/rule/903-commander.html)."

Add a foil information per card

There should be foil information on a per card level. Early sets had no foils, later sets had foils, some sets like Planeshift had just a few cards that could be foil and some sets are foil only.

Right now there is no way to identify using solely the MTGJSON data which sets/cards are available in foil format so this information would be useful.

WPN and Gateway should be split

WPN and Gateway were two separate programs, despite magiccards.info combining them into one. Yes, WPN replaced Gateway, but they are not the same program and right now as they are combined there is no way to determine which set each card is from as the years overlap.

So I should manually figure out which ones were part of which and separate them into pWPN and pGTW

Add deck card restriction info

Some of the duels decks have different restrictions as to how many cards of each rarity can be put into deck, namely: common-4, uncommon-3, rare-2, mythic/planeswalker-1.

Once I do Issue #53 I should also look into adding this 'deck card restriction' info too. Will need to figure out what it all is though. Also, maybe it doesn't belong, because the restriction may be more about the 'format' rather than the 'set'. Will need to see what my research shows when the time comes.

Adding In "Number" to AllSets.json

I see how and why you don't include the number field in the AllSets, but maybe it could be possible to include it.

Seeing as how the arrays printingCodes and printings are in the same order, is it possible to add a number array that follows the same format, holding the same object type as originally designated in the docs? In this case, the values in number are Strings, as they would be in the single set files.

Fictitious Example:

Variable Value
printingCodes [ "ICE", "CHR" ]
printings [ "Ice Age", "Chronicles" ]
number ["3", "185"]

If the array idea above could work, then it could also potentially be applied to the other missing fields: artist, border, flavor, multiverseid, originalText, originalType, rarity, releaseDate, reserved, timeshifted, variations, and watermark

I'd really like to hear your opinion on this ⛵

Problem with mhash

I'm trying to get the mtgjson application to work and for some reason it's throwing an error.

root@farragut:/var/www/isleofgames/mtgjson/mtgjson# node build/buildSet.js allsets

Doing sets: 10E,2ED,3ED,4ED,5DN,5ED,6ED,7ED,8ED,9ED,ALA,ALL,APC,ARB,ARC,ARN,ATQ,AVR,BNG,BOK,BRB,BTD,C13,C14,CHK,CHR,CM1,CMD,CNS,CON,CSP,DD2,DDC,DDD,DDE,DDF,DDG,DDH,DDI,DDJ,DDK,DDL,DDM,DDN,DGM,DIS,DKA,DRB,DRK,DST,EVE,EVG,EXO,FEM,FUT,GPT,GTC,H09,HML,HOP,ICE,INV,ISD,JOU,JUD,KTK,LEA,LEB,LEG,LGN,LRW,M10,M11,M12,M13,M14,M15,MBS,MD1,ME2,ME3,ME4,MED,MIR,MMA,MMQ,MOR,MRD,NMS,NPH,ODY,ONS,PC2,PCY,PD2,PD3,PLC,PLS,PO2,POR,PTK,RAV,ROE,RTR,S00,S99,SCG,SHM,SOK,SOM,STH,THS,TMP,TOR,TSB,TSP,UDS,UGL,ULG,UNH,USG,V09,V10,V11,V12,V13,V14,VAN,VIS,VMA,WTH,WWK,ZEN

Ripping Set: Tenth Edition
Getting card list...
Error ripping: Tenth Edition
TypeError: undefined is not a function
at Object.generateCacheFilePath (/var/www/isleofgames/mtgjson/mtgjson/shared/shared.js:357:16)
at getURLAsDoc (/var/www/isleofgames/mtgjson/mtgjson/build/rip.js:466:25)
at Function.getListHTML (/var/www/isleofgames/mtgjson/mtgjson/build/rip.js:46:4)
at next (/var/www/isleofgames/mtgjson/mtgjson/node_modules/tiptoe/index.js:48:22)
at tiptoe (/var/www/isleofgames/mtgjson/mtgjson/node_modules/tiptoe/index.js:111:2)
at Object.ripSet (/var/www/isleofgames/mtgjson/mtgjson/build/rip.js:24:2)
at Function.build (/var/www/isleofgames/mtgjson/mtgjson/build/buildSet.js:36:8)
at next (/var/www/isleofgames/mtgjson/mtgjson/node_modules/tiptoe/index.js:48:22)
at tiptoe (/var/www/isleofgames/mtgjson/mtgjson/node_modules/tiptoe/index.js:111:2)
at CBRunner.fun (/var/www/isleofgames/mtgjson/mtgjson/build/buildSet.js:33:2)

Line 357 in shared.js is:
var urlHash = hash("whirlpool", targetUrl);

I've confirmed that targetUrl is a valid string, so there seems to be something about the hash function that is throwing the error. I even tried to bring down your mhash project code and rebuild it but it had no effect. What am I doing wrong?

originalCardText has invalid activation costs in earlier cards

Many of the early 4th edtion and earlier cards have messed up activation costs in the 'originalCardText' field (such as '1oRooR' on gatherer for 1RR) and also missing colons for almost all of them. These are gatherer errors that I'll need to correct by hand and add to SET_CORRECTIONS (though some may be able to be fixed in an automated fashion if I can determine a pattern to the errors)

Being that these errors are in the originalCardText and not in the oracle 'text' the priority on fixing this issue is much lower

Move tokens to a new 'tokens' array

As per Rule 108.2 tokens are not considered cards. Thus I should move them out of the 'cards' array and add a new 'tokens' array along side it. The easiest way to do this would probably be AFTER all processing on the set and cards is done, simply move any card where layout==="token" out of the cards array and into the new tokens array.

zip path

AllSets.json.zip unzips into mnt/compendium/DevLab/mtgjson/web/json/AllSets.json. would be nice if it gave just the json file, without the nested directories

Creatures have type "Summon" in some sets

This isn't so important, but... a few cards in pCEL have 'types': ["Summon"]... shouldn't this be ["Creature"]? By the way, the same is true for creatures in Unglued (and one in Unhinged, "Old Fogey" :-). The 'type' attribute has the same issue.

I realize these are special sets, so maybe the usual "normalization" doesn't apply here?

[Gatherer] Fate Reforged watermark errors

Gatherer is wrong about something. 😏

FRF Abzan cards think they have the Temur watermark.
FRF Temur cards think they have no watermark.


I don't suppose you've ever considered sending your ever-growing list of corrections directly to rules manager Matt Tabak?

Several promo sets have invalid number fields

Go through each 'promo' set (where isMCISet is set to true) and 'verify' whether or not the cards have card numbers and 'remove' the 'number' field from any promo sets that shouldn't have a number field. Verify against card images

Update for M15

Just a request to update for the newly spoiled M15 :)
Good job, you guys, keep it up! (tell me if there is something I can do)

Commander 14 is up

Whenever you get a chance, can you add the Commander '14 product (C14)? :)

CON.json invalid file handle

Looks like the json starts many characters into the file and is missing a good chunk of information including the opening json braces.

empty colors

colorless cards have their color value represented in two different ways: as an empty array (the majority), or as a non-existent key (eg ghostfire)

Add 'deck' or 'subset' field

This can be used to identify which cards are part of which decks (such as 'elf' deck in the 'elves vs goblins' set). The existing 'source' field is used a bit like this for some promos sets like CPK, but it's not quite the same thing, but it probably would work just fine for this.

Add additional Duels sets

Should add the additional Duels of the Planeswalkers sets (2012/1013/etc) and other 'Duels' digital sets, such as 'Magic Duels: Origins'. Will need to make sure I get the 'booster' property set correctly as it differs significantly from physical MtG.

Some language's name are not included in AllCards

Is this a Gatherer or script problem?

For example, Mana Crypt has no 'foreignLanguages' array even thought there is a pt-BR version of it and Scavenging Ooze has the pt-BR name in the 'portuguese' index, though there is no 'portuguese (Brazil)' index.

Thanks for the awesome work.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.