Commit graph

1351 commits

Author SHA1 Message Date
Andrew Ekstedt
84f88bacc1 Bump our maximum Markdown version up to 2.6.11
The next commit will fix the warnings, and we need a newer version for
Python 3.9 support because 2.4.1 doesn't work with it.

Markdown 2.6.11 was the last release before 3.0.

Updates #257
2021-03-21 15:39:02 -07:00
Andrew Ekstedt
72f31bc9d3 workflows: Add github actions file
Should be basically the same as the Travis CI file except more verbose
(ugh) and a different version matrix.

- 2.7 3.4 3.5 3.6 3.7 pypy pypy3
+ 2.7 3.5 3.7 3.9 pypy3

Kept 2.7 because we still run it. Dropped pypy2 because who cares?

Dropped 3.4 because it's way old at this point. Added 3.9 because it's
the latest release. Kept 3.5 and 3.7 and dropped the even releases
because it seems silly to test five different releases - the oldest and
newest we support should be adequate (plus 3.7 because that's what i
have.)
2021-03-21 15:39:02 -07:00
Andrew Ekstedt
0a233ec795 Set shape_id for new pokemon to NULL instead of 0
Fixes postgres import.
2021-03-21 15:31:33 -07:00
Andrew Ekstedt
63ac3a8d80
Merge pull request #327 from Parnassius/letsgo-swsh-2
Add Let's Go Pikachu/Eevee and Sword/Shield data

Here is a summary of the extracted data:
- Moves (names, flavor text, changelogs, flags, effects)
- Abilities (names, flavor text)
- Items (names, game indices, machines)
- Pokemon (names, species, formes, learnsets, egg groups, dex numbers, dex entries, evolutions, categories (genus))

Locations and encounters are not included.

The Let's Go data excludes unobtainable pokemon; i.e., everything except
the first 151 and their forms, plus Meltan and Melmetal. While the game
has data for the rest of the pokemon, they are explicitly unobtainable
even by trading, and the data is mostly (or entirely?) copied from Sun &
Moon.

TMs and TRs are numbered from 00 to 99 in Sword/Shield. We use machine
numbers 0-99 for the TMs, and 100-199 for the TRs. For example TM05 is ID
5, and TR99 is ID 199. Note that TRs overlap with HMs in previous games.

Friendship-based evolutions now trigger at 160 friendship instead of 220.

Pokemon shapes are left blank (it's unclear if they are in Sw/Sh at all).
We can grab them from HOME later. Other blank columns: pokemon_name (in
pokemon_form_names), is_battle_only, has_gender_differences, and form_switchable.

Parnassius's python scripts for extracting the game data are available
here: <https://github.com/Parnassius/i-pokemoni>.
They are heavily based on the pknx project:
<https://github.com/kwsch/pkNX>.

Closes #311
Closes #284
Closes #262
Closes #210
Closes #231
2021-03-21 15:22:48 -07:00
Andrew Ekstedt
5c87154d4c Require SQLAlchemy <1.4
Temporary fix for #330
2021-03-18 20:52:14 -07:00
Andrew Ekstedt
e9bc81707e Make Pokemon.shape_id nullable 2021-03-14 18:12:58 -07:00
Andrew Ekstedt
de27f2c282 Fix test for missing default forms 2021-03-14 18:12:58 -07:00
Parnassius
1cc1611da3 item categories 2021-03-10 14:27:54 +01:00
Parnassius
67d9e8a9f9 Add Sword/Shield data 2021-03-10 14:25:32 +01:00
Parnassius
ebf2fb0cdd item categories 2021-03-10 14:22:00 +01:00
Parnassius
0ae34dbce0 Add Let's Go data 2021-03-10 14:21:05 +01:00
Andrew Ekstedt
a91a7d95b3 Allow small ッ before a vowel
This is a nonstandard use of ッ and it doesn't really have a defined
romanization, but we need to support it for Cramorant (ウッウ, U'u).
2021-03-07 11:26:50 -08:00
Andrew Ekstedt
b09929dfb6 add scripts/rename-home.py
My oneshot script for renaming the HOME sprites from the web rip. In
case we need it in the future for re-ripping, and to record the form
identifiers i used since the new ones aren't in the DB yet.
2020-09-03 12:58:31 -07:00
Andrew Ekstedt
dc9a2d8f13 Fix Spanish characteristic text
"Quick tempered" and "A little quick tempered" erroneously had the same
Spanish translation, "Tiene mal genio". The latter should actually be
"A veces se enfada".

Verified against a text dump.

Fixes #309
2020-07-09 12:39:16 -07:00
Andrew Ekstedt
9397da82c2 Update Partner Cap Pikachu's combined form name 2020-06-27 16:33:42 -07:00
Andrew Ekstedt
8f1dc80d12 Rip OR/AS held items
Note: both of Cherrim's forms hold a Miracle Seed, but we only have one
pokemon row for them.

Updates #141
2020-06-27 16:31:33 -07:00
Andrew Ekstedt
9e15919f14 Fix typo in Totem Kommo-o's form name
Why do we even have totem pokemon in the dex?

Fixes #222
2020-06-27 13:05:14 -07:00
KidsDontPlay
7de90c054d Add is_legendary and is_mythical (#304)
Legendary and Mythical are official terms which, while generally not
present in the game data, are more or less consistently applied in
promotional material. They're also useful categorizations to fans,
so it seems reasonable to include them here.

The legendary Pokémon are:
  Articuno, Zapdos, Moltres, Mewtwo,
  Raikou, Entei, Suicune, Lugia, Ho-Oh,
  Regirock, Regice, Registeel,
  Latias, Latios, Kyogre, Groudon, Rayquaza,
  Uxie, Mesprit, Azelf, Dialga, Palkia, Giratina,
  Heatran, Regigigas, Cresselia,
  Cobalion, Terrakion, Virizion,
  Tornadus, Thundurus, Landorus,
  Reshiram, Zekrom, Kyurem,
  Xerneas, Yveltal, Zygarde,
  Tapu Koko, Tapu Lele, Tapu Bulu, Tapu Fini,
  Cosmog, Cosmoem, Solgaleo, Lunala, Necrozma.

The mythical Pokémon are: Mew, Celebi, Jirachi, Deoxys, Phione, Manaphy,
  Darkrai, Shaymin, Arceus, Victini, Keldeo, Meloetta, Genesect,
  Diancie, Hoopa, Volcanion, Magearna, Marshadow, Zeraora.
2020-06-27 12:56:28 -07:00
Andrew Ekstedt
bbda228262 Fix み,nbsp mixup in OR/AS dex entries
A bug in the text dumper caused み to be replaced with a narrow
non-breaking space and the actual nbsp character in the unicode private
use area to go untranslated.

Only affected OR/AS.

Fix with a simple text replace on the affected entries.

    import csv
    in_ = open("../pokedex/data/csv/pokemon_species_flavor_text.csv", "r")
    out = open("fixed.csv", "w")
    r = csv.reader(in_, lineterminator='\n')
    w = csv.writer(out, lineterminator='\n')
    for row in r:
        text = row[3]
        if row[1] in {'25', '26'}:
            text = text.replace("\u202f", "\u307f") # nbsp => mi
            text = text.replace("\ue07f", "\u202f") # e07f => nbsp
        w.writerow((row[0], row[1], row[2], text))
    out.close()

Fixes #308
2020-06-23 17:09:11 -07:00
Andrew Ekstedt
0015bcb15c Add Ultra Moon dex entries
Note: there are dex entries for forms too, but this only adds the text
for the default form because the db schema assumes dex entries are
per-species.
2020-06-23 16:17:32 -07:00
Andrew Ekstedt
5fe8fdcc95 Add Sun dex entries
Note: there are dex entries for forms too, but this only adds the text
for the default form because the db schema assumes dex entries are
per-species.
2020-06-23 16:10:40 -07:00
Andrew Ekstedt
b05ff3a5b1 Add wild held items for Sun and Ultra Moon
The initial import of data from S/M and US/UM only included held items for Moon
and Ultra Sun. (Presumably those were the only games eevee had access to?)
I now have all four games and I've verified that the data file which
contains held items is completely identical between Sun and Moon, and between
Ultra Sun and Ultra Moon.

So we can just copy the held items from the other two games.

pokedex=> insert into pokemon_items (select pokemon_id, 27, item_id, rarity from pokemon_items where version_id=28);
INSERT 0 356
Time: 220.560 ms
pokedex=> insert into pokemon_items (select pokemon_id, 30, item_id, rarity from pokemon_items where version_id=29);
INSERT 0 362
Time: 102.063 ms
2020-06-22 20:52:01 -07:00
rluzuriaga
f535b61341 Handle CLI load into existing tables gracefully (PR #300)
- Add exception handling for SQLite, PostgreSQL, and MySQL.
Sqlalchemy gives different exceptions for each database engine so I have
seperated each exception for each engine.
The error message passed by each engine is also different so that is
checked as well.
- Once the function gets one of those exceptions the following message
is outputed and the program ends: `ERROR:  Table 'TABLENAME' already
exists in the database. Did you mean to use 'pokedex load -D'`
- If the error is not that a table already exists, then the output will
just be the full error from sqlalchemy. That way if someone runs into
that error it can be reported as an issue and whoever can help would see
the full error.
---
Resolves part of issue #16
2020-06-17 16:12:32 -07:00
Andrew Ekstedt
c2cde0078d Fix typo in Fancy Vivillon's French name
Prismillion Fantaisie => Prismillon Fantaisie (without the extra 'i')

Fixes #307
2020-06-17 15:16:21 -07:00
Andrew Ekstedt
98696cdd69 Fix pokedex search command
Filtering on pokemon name was broken. Not sure why, since the search CLI
was added way after the i18n stuff was added. The error is related to
AssociationProxy, which figures because nothing about association
proxies ever seems to work right. I don't know enough about SQLAlchemy
internals to know if what it was trying to do was supposed to
work, or how to fix it if so. So, fix it by using the same boring
join-based filtering that spline-pokedex uses.

Running `pokedex search --name=gloom` caused the following error with
PostgreSQL and SQLAlchemy 1.3.5:

    sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'ColumnAssociationProxyInstance'

...and this error with SQLite.

    sqlalchemy.exc.InterfaceError: (sqlite3.InterfaceError) Error binding parameter 4 - probably unsupported type.
    [parameters: (9, 9, 9, 9, ColumnAssociationProxyInstance(AssociationProxy('names_local', 'name')), 'gloom')]

Additionaly, the following error would happen with PostgreSQL and SQLAlchemy 0.9.7,
but that's probably unrelated:

    sqlalchemy.exc.ProgrammingError: (ProgrammingError) subquery in FROM must have an alias
    LINE 4: FROM pokemon_species, (SELECT pokemon_species_names.name AS ...
                                   ^

Fixes #296
2020-04-17 16:42:47 -07:00
Andrew Ekstedt
e5c18c4109 Travis CI: check that dump is idempotent
Run git-diff after dumping the database to make sure that there are no
changes to the csv files.

We often get PRs from people who edit stuff by hand and don't know to do
a load/dump of the database afterwards. Usually this means they were
adding game data by hand, which we want to discourage, but even if it's
done for valid reasons it's good to canonicalize the csv files as soon
as possible to avoid interfering with future commits.
2020-03-04 16:19:29 -08:00
Andrew Ekstedt
77e3d9df16 Travis CI: don't pass -l to dump
The help text for pokedex dump -l says

    -l LANGS, --langs LANGS
                        comma-separated list of language codes to load,
                        'none', or 'all' (default: en)

However, the values 'none' and 'all' don't work (they are treated as
language names) and the default is to dump all languages, not just
English.

So `dump -l all` actually dumps no text at all because there is no
language called 'all'.

This is about to cause trouble for Travis CI because the next commit
will start checking that load/dump successfully round-trips... which
definitely isn't the case right now because the dump is missing
all text from translated tables.

The dump command and its documentation should definitely be fixed at
some point, but for now just stop using -l.
2020-03-04 16:19:29 -08:00
Andrew Ekstedt
a166502b45 Load/dump dance
Fixes the order of encounter_condition_value_map.csv after PR #289.
2020-03-04 16:19:29 -08:00
Andrew Ekstedt
51957b9b37 Travis CI: fix some config warnings
`sudo: false` used to be the way to opt in to their container-based
infrastructure. That system was retired in 2018 and so the `sudo` option
no longer does anything. Remove it.

Travi CI supports multiple OSes now. Add `os: linux` to be explicit
about which OS we want to test on.
2020-03-04 15:27:38 -08:00
Andrew Ekstedt
a0b47add63
Update Spanish translations (#271) 2020-03-04 15:24:28 -08:00
Andrew Ekstedt
9c25b268eb
Update README and improve docker support (#294)
* improve python 3, docker support and README

- fix support for python 3.7
- simplify docker support
- update readme
- remove obsolete item-questions file
2020-03-04 15:07:03 -08:00
Andrew Ekstedt
722a4664dd
Generation 2 (GSC) encounters (#289)
Source: https://www.pokebip.com/page/jeuxvideo/dossier_shasse/encounter_slot

This covers the following encounter methods: Walking, Fishing (3 rods), Surf and Rock Smash
2020-03-04 14:52:07 -08:00
Andrew Ekstedt
9c1d1828c9 Re-rip Gen III contest effects
Gen III contest effects were part of the initial data import from brown
veekun and so have probably never gotten a proper rip.

This commit re-rips contest effect stats, types, combos, and the mapping
between contest effects and moves.

None of the stats changed (jam & appeal); a few moves got assigned
different contest effects (specifically: smokescreen, flash, spite, and
shadow ball); and several combos changed.

For example, Hail was listed as comboing into the following moves
(this is also what Bulbapedia says):

    Ice Beam, Blizzard, Aurora Beam, Haze, Powder Snow, Icy Wind,
    Ice Ball, Weather Ball, Sheer Cold, Icicle Spear

However, the actual list is much shorter:

    Blizzard, Powder Snow, Weather Ball

I verified on my Ruby cart that Hail->Ice Beam and Hail->Icy Wind are
not real combos, and that Hail->Blizzard is.

n.b. There are a bunch of unused contest effects in the game: out of 48
effects, 15 are not used by any move. The pokedex doesn't currently
include these. I elected not to add them, partly because i didn't want
to have to rip flavor text for them, but mostly because if i were going
to go that route i would probably reassign contest_effects.id to match
their internal ids, which would have made for uglier diffs.

Fixes #280
2020-03-01 23:25:49 -08:00
KidsDontPlay
cc2ad50752
Fix typo in German name for Physical damage class (#283) 2020-03-01 22:53:02 -08:00
Andrew Ekstedt
0232456433 Load/dump dance
Fixes some out-of-order rows in pokemon_dex_numbers.csv that were
introduced by PR #269 (f8e7637e9).
2020-02-29 20:02:08 -08:00
Andrew Ekstedt
7d5fbaec87 Fix some more issues in German ability effect text
- link to nonexistant `[]{move:shield}` in Color Change (changed to protect)
- invalid syntax `[move:sandstorm]` in Primordial Sea
- invalid syntax `[move:sandstorm]` in Desolate Land
- invalid syntax `[move:sandstorm]` in Delta Stream

Found by test_markdown_values, which is not run by default because it is
slow. It can be run manually with

    py.test pokedex/tests/test_strings.py --all
2020-01-23 12:47:01 -08:00
Andrew Ekstedt
e78f7cbe84 Fix markdown syntax in Flower Veil's German effect text 2020-01-23 12:05:36 -08:00
Alessandro Pezzè
049fa1437b Fix markdown syntax in Cheek Pouch's German effect text (#290) 2020-01-22 19:12:40 -08:00
Andrew Ekstedt
03974e6f22 Travis CI: test Python 3.6 and 3.7 2019-11-15 22:35:12 -08:00
Andrew Ekstedt
8dfbd58688 Regenerate order columns
Updates #282
2019-11-15 22:13:33 -08:00
Andrew Ekstedt
852a4ad06a Fix form_order for arceus-unknown and pikachu-*-cap
Pikachu's Cap forms and Cosplay forms had overlapping orders because i
forgot about Cosplay Pikachu when i set the form_order for the cap
forms.

??? Arceus had form_order=3 so it would go after all the other forms,
which had form_order=2. These were assigned distinct form_orders 2
through 17 at some point during a recent dump, but ??? Arceus was
missed.

Fixes #282
2019-11-15 22:01:12 -08:00
Zroxos
3c550f1425 Add evolution method to Crabrawler (#276)
Added missing `location_id` for Crabrawler. It now references the "Mount Lanakila" row in the `locations` table.

Updates #248
2019-07-21 22:42:19 -07:00
h82258652
087237c9ef Fix US/UM chinese pokemon names (#275)
The Chinese names for some of the pokemon added in US/UM (805-807) seem to have gotten corrupted somehow.
2019-07-21 22:41:02 -07:00
naooooooo99999
310d104533 Fixed the spelling of japanese Generation I. (#267) 2019-07-16 23:26:54 -07:00
ymin1103
ce783fba03 Add Korean translations and missing version names (#266)
This commit adds Korean translations for egg groups, generations, languages, regions, stats, and version names. Also the ??? and shadow types. 

It also fills in some missing foreign version names for S/M and US/UM (from Bulbapedia).
2019-07-16 23:18:02 -07:00
kampfmoehre
10f024a80c Add german ability translations (#181)
Adds german translations for ability description prose and ability
changelog prose. I tried to stay in the same style for all translations.
2019-07-16 22:39:18 -07:00
Andrew Ekstedt
ad429ab128 Fix Roserade's Central Kalos dex number and add tests
Roserade and Roselia were mistakenly sharing dex number 72. Presumably
an error from the original X/Y spreadsheet. I guess we never reripped
these after 3DS rom dumping became possible.

Regardless of how it happened, add a test and some database constraints
to ensure that it doesn't happen again.

- Add a test for gaps in pokedex numbers

- Add uniqueness constraints to pokemon_dex_numbers.  A species can
  only appear once per pokedex, and a number cannot be used more than
  once per pokedex.
2019-07-16 18:12:58 -07:00
Tim Goudriaan
f8e7637e9e Add S/M and US/UM pokedex numbers (#269)
Adds pokedex numbers for the Alola dex and all four island dexes, both original and updated.

Data courtesy of Bulbapedia.

Fixes #217.
2019-07-16 17:52:01 -07:00
Kip Yin
11f972f817 Catch StopIteration explicitly in group_by_object generator (#264)
Python 3.7 changed the behaviour of generators so that a StopIteration exception which bubbles up inside a generator is transformed into a RuntimeException. This means that when calling next() inside a generator, we have to explicitly catch the StopIteration and exit the function instead of relying on it to implicitly stop the generator. Otherwise the program will crash. Sigh.

For more information, see PEP 479.

https://www.python.org/dev/peps/pep-0479/
https://stackoverflow.com/questions/51700960/runtimeerror-generator-raised-stopiteration-every-time-i-try-to-run-app/51701040#51701040
2019-07-15 13:23:37 -07:00
Kip Yin
c63b6522bf Limit construct version to 2.5.3
This project uses `construct` 2.5 and lower, and the latest `construct` version (2.9)
is incompatible with 2.5.

Resolves #265
2019-07-15 13:04:05 -07:00