Commit Graph

1376 Commits

Author SHA1 Message Date
Andrew Ekstedt a15df35f29 Fix Violet Meteor Minior's name
Reported by Hollace
2021-03-26 18:28:09 -07:00
Andrew Ekstedt 605ebf61de Update Alolan form names
Sword/Shield changed the english form name of all Alolan pokemon from
"Alola Form" to "Alolan Form" (other languages unchanged). This commit
makes the same change for pokemon which are not in Sword or Shield.
2021-03-26 13:29:29 -07:00
Andrew Ekstedt 54c6c7faa8 Name Totem Alolan Marowak 2021-03-26 00:05:23 -07:00
Andrew Ekstedt 9557034bad Add form names for Eternamax Eternatus
Copied manually from a text dump.
2021-03-26 00:04:40 -07:00
Andrew Ekstedt b8ff7b4cb5 Add form names for Crowned Zacian
Copied manually from a text dump. Not sure why these were missing.
2021-03-26 00:04:40 -07:00
Andrew Ekstedt 0bc0953578 Add combined form names for new Gen 8 pokemon 2021-03-26 00:04:40 -07:00
Andrew Ekstedt 3163bab947 Drop empty rows from pokemon_form_names
A missing name means the same thing as a blank name, so there's no point
keeping them around.

Totem Marowak has a Korean form name for some reason, and Kyogre and
Groudon have (autoeponymous) form names in every language _except_
Italian. But other than that there's nothing interesting.
2021-03-26 00:01:11 -07:00
Andrew Ekstedt 9d125771b3 Update version_group_pokemon_move_methods for Let's Go and Sw/Sh 2021-03-25 00:32:11 -07:00
Andrew Ekstedt ba0409362a
Merge pull request #332 from Parnassius/fix-zygarde
zygarde-10 was added to the database along with the Let's Go rip, but the form was actually introduced in sun/moon. We need to backdate it.
2021-03-24 20:22:57 -07:00
Andrew Ekstedt ffff1ee0b7
Merge pull request #331 from Parnassius/gen7-tutors
Dump US/UM tutors

[Tutors][1] were taken from [pk3DS][2]). In both generation 7 games there are 3 additional tutors, which afaik are simply hardcoded (pikachu => volt tackle, keldeo => secret sword, meloetta => relic song), so I added them [by hand][3].

[1]: 46ab34c44f/scripts/dump_usum.py (L50):L130
[2]: 754b9b8d95/pk3DS/Subforms/Gen7/PersonalEditor7.cs (L63):L73
[3]: https://github.com/Parnassius/i-pokemoni/blob/master/scripts/dump_usum.py#L131:L137

Since i dumped the learnsets as a whole, this also adds a couple of other things:
- egg moves for non-default pokemon formes, which were previously missing (see #219, i plan to dump them from ORAS as well soon-ish)
- the learnsets for the zygarde form added in #327
- the `order` column for evolution moves (level 0), which was previously missing

Closes #223 
Closes #241
2021-03-24 20:01:52 -07:00
Parnassius 03f32883f2 zygarde-10 was introduced in sun/moon 2021-03-24 21:40:07 +01:00
Parnassius ea6d516333 Dump UsUm tutors 2021-03-24 17:33:25 +01:00
Parnassius a83e44f732 Dump SM tutors 2021-03-24 17:31:23 +01:00
Parnassius 80bc35a764 Remove HMs from Sun/Moon and Ultra Sun/Ultra Moon (PR #326)
Those games don't have HMs, and for example
https://veekun.com/dex/moves/fly shows fly is HM02, while it should be
TM76. Note that HMs which were converted into TMs in gen7 seems to be
already present in machines.csv with their new TM number as well.
2021-03-21 17:06:12 -07:00
Andrew Ekstedt 47618d0d46
Merge pull request #329 from SnorlaxMonster/area-rename
Rename Mt. Coronet areas

From the PR:
> Mt. Coronet's floors were misnumbered: a small room on 4F was labelled as 5F, causing the subsequent floors to all be off-by-one. This pull request corrects that numbering, aligning the floor numbering with the official guidebook.
> 
> Additionally, a number of the areas within Mt. Coronet had vague or otherwise unclear names (especially "cave"). This pull request renames those areas for clarity. It also ensures that all areas on 1F include "1F" in their name, for consistency.
2021-03-21 16:59:23 -07:00
Andrew Ekstedt 6d81374fb1 Fix machines<->items mapping for Sw/Sh
They were cyclically off by one, so e.g. machine 0 mapped to tm01,
machine 99 wrapped around to tm00, machine 100 mapped to tr01, and
machine 199 mapped to tr00.

The moves seem to be right though.

update machines m set item_id = (select item_id from machines m2 where m2.machine_number = (m.machine_number+99) % 100 + case when m.machine_number >= 100 then 100 else 0 end and version_group_id = 20) where version_group_id = 20;
2021-03-21 16:49:00 -07:00
Andrew Ekstedt dcda895e22 Merge branch 'gh-actions' 2021-03-21 15:52:12 -07:00
Andrew Ekstedt 7630d2c7ba Update link to github actions docs 2021-03-21 15:46:56 -07:00
Andrew Ekstedt 3eb213aab4 Update list of supported python versions 2021-03-21 15:39:02 -07:00
Andrew Ekstedt f244234248 Register pytest.mark.slow marker
Pytest started warning about unknown markers at some point. Not sure
when. My local version doesn't do it but the CI versions do.

> PytestUnknownMarkWarning: Unknown pytest.mark.slow - is this a typo?
  You can register custom marks to avoid this warning - for details, see
  https://docs.pytest.org/en/stable/mark.html
2021-03-21 15:39:02 -07:00
Andrew Ekstedt 2faa3ed6af Avoid a warning about regex flags in Python 3.6
Python 3.6 deprecated using regex flag syntax (?x) to set flags in the
middle of a pattern. Now you can only use it at the start of a pattern.
Fortunately that same release added a new scoped-flag syntax, (?x:).

(Wait, you say, it looks like our code *does* set flags at the start of
the pattern? That's true, but the markdown module includes our regex in the
middle of a larger one, so it's not actually at the start.)

Fixes this warning (and a couple similar ones):

DeprecationWarning: Flags not at the start of the expression '^(.?)(?x) \[ ([^]]' (truncated)
    self.compiled_re = re.compile(r"^(.?)%s(.)$" % pattern
2021-03-21 15:39:02 -07:00
Andrew Ekstedt 348742fe2a Close csv files after dumping
PyPy doesn't collect garbage as quickly as CPython, so it was possible
for the process to exit before all the files had been flushed.
2021-03-21 15:39:02 -07:00
Andrew Ekstedt 68ee98ffdd Fix some Markdown warnings
Fixes the following warnings when run with Markdown 2.6.11:

DeprecationWarning: "safe_mode" is deprecated in Python-Markdown. Use an HTML sanitizer (like Bleach https://bleach.readthedocs.io/) if you are parsing untrusted markdown text. See the 2.6 release notes for more info
DeprecationWarning: Using short names for Markdown's builtin extensions is deprecated. Use the full path to the extension with Python's dot notation (eg: "markdown.extensions.extra" instead of "extra"). The current behavior will raise an error in version 2.7. See the Release Notes for Python-Markdown version 2.6 for more info.

Also get rid of some compatability code for Markdown 2.0, which we no longer support.

Updates #257
2021-03-21 15:39:02 -07:00
Andrew Ekstedt 84f88bacc1 Bump our maximum Markdown version up to 2.6.11
The next commit will fix the warnings, and we need a newer version for
Python 3.9 support because 2.4.1 doesn't work with it.

Markdown 2.6.11 was the last release before 3.0.

Updates #257
2021-03-21 15:39:02 -07:00
Andrew Ekstedt 72f31bc9d3 workflows: Add github actions file
Should be basically the same as the Travis CI file except more verbose
(ugh) and a different version matrix.

- 2.7 3.4 3.5 3.6 3.7 pypy pypy3
+ 2.7 3.5 3.7 3.9 pypy3

Kept 2.7 because we still run it. Dropped pypy2 because who cares?

Dropped 3.4 because it's way old at this point. Added 3.9 because it's
the latest release. Kept 3.5 and 3.7 and dropped the even releases
because it seems silly to test five different releases - the oldest and
newest we support should be adequate (plus 3.7 because that's what i
have.)
2021-03-21 15:39:02 -07:00
Andrew Ekstedt 0a233ec795 Set shape_id for new pokemon to NULL instead of 0
Fixes postgres import.
2021-03-21 15:31:33 -07:00
Andrew Ekstedt 63ac3a8d80
Merge pull request #327 from Parnassius/letsgo-swsh-2
Add Let's Go Pikachu/Eevee and Sword/Shield data

Here is a summary of the extracted data:
- Moves (names, flavor text, changelogs, flags, effects)
- Abilities (names, flavor text)
- Items (names, game indices, machines)
- Pokemon (names, species, formes, learnsets, egg groups, dex numbers, dex entries, evolutions, categories (genus))

Locations and encounters are not included.

The Let's Go data excludes unobtainable pokemon; i.e., everything except
the first 151 and their forms, plus Meltan and Melmetal. While the game
has data for the rest of the pokemon, they are explicitly unobtainable
even by trading, and the data is mostly (or entirely?) copied from Sun &
Moon.

TMs and TRs are numbered from 00 to 99 in Sword/Shield. We use machine
numbers 0-99 for the TMs, and 100-199 for the TRs. For example TM05 is ID
5, and TR99 is ID 199. Note that TRs overlap with HMs in previous games.

Friendship-based evolutions now trigger at 160 friendship instead of 220.

Pokemon shapes are left blank (it's unclear if they are in Sw/Sh at all).
We can grab them from HOME later. Other blank columns: pokemon_name (in
pokemon_form_names), is_battle_only, has_gender_differences, and form_switchable.

Parnassius's python scripts for extracting the game data are available
here: <https://github.com/Parnassius/i-pokemoni>.
They are heavily based on the pknx project:
<https://github.com/kwsch/pkNX>.

Closes #311
Closes #284
Closes #262
Closes #210
Closes #231
2021-03-21 15:22:48 -07:00
Andrew Ekstedt 5c87154d4c Require SQLAlchemy <1.4
Temporary fix for #330
2021-03-18 20:52:14 -07:00
Andrew Ekstedt e9bc81707e Make Pokemon.shape_id nullable 2021-03-14 18:12:58 -07:00
Andrew Ekstedt de27f2c282 Fix test for missing default forms 2021-03-14 18:12:58 -07:00
Parnassius 1cc1611da3 item categories 2021-03-10 14:27:54 +01:00
Parnassius 67d9e8a9f9 Add Sword/Shield data 2021-03-10 14:25:32 +01:00
Parnassius ebf2fb0cdd item categories 2021-03-10 14:22:00 +01:00
Parnassius 0ae34dbce0 Add Let's Go data 2021-03-10 14:21:05 +01:00
SnorlaxMonster 06aaf1b9aa Improve Mt. Coronet area names
The Mt. Coronet area names were not sufficiently clear as to which
areas they were referring to (especially the one simply named "cave").
I have renamed all of the areas that I think were unclear,
as well as ensured that all 1F areas in Mt. Coronet
include "1F" in their name for consistency.
2021-03-09 02:18:14 +11:00
SnorlaxMonster 437736ca01 Correct a floor numbering error in Mt. Coronet
The small room on Mt. Coronet 4F was labelled as 5F,
causing all subsequent floors to be numbered off-by-one.
2021-03-09 02:08:28 +11:00
Andrew Ekstedt a91a7d95b3 Allow small ッ before a vowel
This is a nonstandard use of ッ and it doesn't really have a defined
romanization, but we need to support it for Cramorant (ウッウ, U'u).
2021-03-07 11:26:50 -08:00
Andrew Ekstedt b09929dfb6 add scripts/rename-home.py
My oneshot script for renaming the HOME sprites from the web rip. In
case we need it in the future for re-ripping, and to record the form
identifiers i used since the new ones aren't in the DB yet.
2020-09-03 12:58:31 -07:00
Andrew Ekstedt dc9a2d8f13 Fix Spanish characteristic text
"Quick tempered" and "A little quick tempered" erroneously had the same
Spanish translation, "Tiene mal genio". The latter should actually be
"A veces se enfada".

Verified against a text dump.

Fixes #309
2020-07-09 12:39:16 -07:00
Andrew Ekstedt 9397da82c2 Update Partner Cap Pikachu's combined form name 2020-06-27 16:33:42 -07:00
Andrew Ekstedt 8f1dc80d12 Rip OR/AS held items
Note: both of Cherrim's forms hold a Miracle Seed, but we only have one
pokemon row for them.

Updates #141
2020-06-27 16:31:33 -07:00
Andrew Ekstedt 9e15919f14 Fix typo in Totem Kommo-o's form name
Why do we even have totem pokemon in the dex?

Fixes #222
2020-06-27 13:05:14 -07:00
KidsDontPlay 7de90c054d Add is_legendary and is_mythical (#304)
Legendary and Mythical are official terms which, while generally not
present in the game data, are more or less consistently applied in
promotional material. They're also useful categorizations to fans,
so it seems reasonable to include them here.

The legendary Pokémon are:
  Articuno, Zapdos, Moltres, Mewtwo,
  Raikou, Entei, Suicune, Lugia, Ho-Oh,
  Regirock, Regice, Registeel,
  Latias, Latios, Kyogre, Groudon, Rayquaza,
  Uxie, Mesprit, Azelf, Dialga, Palkia, Giratina,
  Heatran, Regigigas, Cresselia,
  Cobalion, Terrakion, Virizion,
  Tornadus, Thundurus, Landorus,
  Reshiram, Zekrom, Kyurem,
  Xerneas, Yveltal, Zygarde,
  Tapu Koko, Tapu Lele, Tapu Bulu, Tapu Fini,
  Cosmog, Cosmoem, Solgaleo, Lunala, Necrozma.

The mythical Pokémon are: Mew, Celebi, Jirachi, Deoxys, Phione, Manaphy,
  Darkrai, Shaymin, Arceus, Victini, Keldeo, Meloetta, Genesect,
  Diancie, Hoopa, Volcanion, Magearna, Marshadow, Zeraora.
2020-06-27 12:56:28 -07:00
Andrew Ekstedt bbda228262 Fix み,nbsp mixup in OR/AS dex entries
A bug in the text dumper caused み to be replaced with a narrow
non-breaking space and the actual nbsp character in the unicode private
use area to go untranslated.

Only affected OR/AS.

Fix with a simple text replace on the affected entries.

    import csv
    in_ = open("../pokedex/data/csv/pokemon_species_flavor_text.csv", "r")
    out = open("fixed.csv", "w")
    r = csv.reader(in_, lineterminator='\n')
    w = csv.writer(out, lineterminator='\n')
    for row in r:
        text = row[3]
        if row[1] in {'25', '26'}:
            text = text.replace("\u202f", "\u307f") # nbsp => mi
            text = text.replace("\ue07f", "\u202f") # e07f => nbsp
        w.writerow((row[0], row[1], row[2], text))
    out.close()

Fixes #308
2020-06-23 17:09:11 -07:00
Andrew Ekstedt 0015bcb15c Add Ultra Moon dex entries
Note: there are dex entries for forms too, but this only adds the text
for the default form because the db schema assumes dex entries are
per-species.
2020-06-23 16:17:32 -07:00
Andrew Ekstedt 5fe8fdcc95 Add Sun dex entries
Note: there are dex entries for forms too, but this only adds the text
for the default form because the db schema assumes dex entries are
per-species.
2020-06-23 16:10:40 -07:00
Andrew Ekstedt b05ff3a5b1 Add wild held items for Sun and Ultra Moon
The initial import of data from S/M and US/UM only included held items for Moon
and Ultra Sun. (Presumably those were the only games eevee had access to?)
I now have all four games and I've verified that the data file which
contains held items is completely identical between Sun and Moon, and between
Ultra Sun and Ultra Moon.

So we can just copy the held items from the other two games.

pokedex=> insert into pokemon_items (select pokemon_id, 27, item_id, rarity from pokemon_items where version_id=28);
INSERT 0 356
Time: 220.560 ms
pokedex=> insert into pokemon_items (select pokemon_id, 30, item_id, rarity from pokemon_items where version_id=29);
INSERT 0 362
Time: 102.063 ms
2020-06-22 20:52:01 -07:00
rluzuriaga f535b61341 Handle CLI load into existing tables gracefully (PR #300)
- Add exception handling for SQLite, PostgreSQL, and MySQL.
Sqlalchemy gives different exceptions for each database engine so I have
seperated each exception for each engine.
The error message passed by each engine is also different so that is
checked as well.
- Once the function gets one of those exceptions the following message
is outputed and the program ends: `ERROR:  Table 'TABLENAME' already
exists in the database. Did you mean to use 'pokedex load -D'`
- If the error is not that a table already exists, then the output will
just be the full error from sqlalchemy. That way if someone runs into
that error it can be reported as an issue and whoever can help would see
the full error.
---
Resolves part of issue #16
2020-06-17 16:12:32 -07:00
Andrew Ekstedt c2cde0078d Fix typo in Fancy Vivillon's French name
Prismillion Fantaisie => Prismillon Fantaisie (without the extra 'i')

Fixes #307
2020-06-17 15:16:21 -07:00
Andrew Ekstedt 98696cdd69 Fix `pokedex search` command
Filtering on pokemon name was broken. Not sure why, since the search CLI
was added way after the i18n stuff was added. The error is related to
AssociationProxy, which figures because nothing about association
proxies ever seems to work right. I don't know enough about SQLAlchemy
internals to know if what it was trying to do was supposed to
work, or how to fix it if so. So, fix it by using the same boring
join-based filtering that spline-pokedex uses.

Running `pokedex search --name=gloom` caused the following error with
PostgreSQL and SQLAlchemy 1.3.5:

    sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'ColumnAssociationProxyInstance'

...and this error with SQLite.

    sqlalchemy.exc.InterfaceError: (sqlite3.InterfaceError) Error binding parameter 4 - probably unsupported type.
    [parameters: (9, 9, 9, 9, ColumnAssociationProxyInstance(AssociationProxy('names_local', 'name')), 'gloom')]

Additionaly, the following error would happen with PostgreSQL and SQLAlchemy 0.9.7,
but that's probably unrelated:

    sqlalchemy.exc.ProgrammingError: (ProgrammingError) subquery in FROM must have an alias
    LINE 4: FROM pokemon_species, (SELECT pokemon_species_names.name AS ...
                                   ^

Fixes #296
2020-04-17 16:42:47 -07:00