Commit graph

408 commits

Author SHA1 Message Date
Andrew Ekstedt
348742fe2a Close csv files after dumping
PyPy doesn't collect garbage as quickly as CPython, so it was possible
for the process to exit before all the files had been flushed.
2021-03-21 15:39:02 -07:00
Andrew Ekstedt
68ee98ffdd Fix some Markdown warnings
Fixes the following warnings when run with Markdown 2.6.11:

DeprecationWarning: "safe_mode" is deprecated in Python-Markdown. Use an HTML sanitizer (like Bleach https://bleach.readthedocs.io/) if you are parsing untrusted markdown text. See the 2.6 release notes for more info
DeprecationWarning: Using short names for Markdown's builtin extensions is deprecated. Use the full path to the extension with Python's dot notation (eg: "markdown.extensions.extra" instead of "extra"). The current behavior will raise an error in version 2.7. See the Release Notes for Python-Markdown version 2.6 for more info.

Also get rid of some compatability code for Markdown 2.0, which we no longer support.

Updates #257
2021-03-21 15:39:02 -07:00
Andrew Ekstedt
e9bc81707e Make Pokemon.shape_id nullable 2021-03-14 18:12:58 -07:00
KidsDontPlay
7de90c054d Add is_legendary and is_mythical (#304)
Legendary and Mythical are official terms which, while generally not
present in the game data, are more or less consistently applied in
promotional material. They're also useful categorizations to fans,
so it seems reasonable to include them here.

The legendary Pokémon are:
  Articuno, Zapdos, Moltres, Mewtwo,
  Raikou, Entei, Suicune, Lugia, Ho-Oh,
  Regirock, Regice, Registeel,
  Latias, Latios, Kyogre, Groudon, Rayquaza,
  Uxie, Mesprit, Azelf, Dialga, Palkia, Giratina,
  Heatran, Regigigas, Cresselia,
  Cobalion, Terrakion, Virizion,
  Tornadus, Thundurus, Landorus,
  Reshiram, Zekrom, Kyurem,
  Xerneas, Yveltal, Zygarde,
  Tapu Koko, Tapu Lele, Tapu Bulu, Tapu Fini,
  Cosmog, Cosmoem, Solgaleo, Lunala, Necrozma.

The mythical Pokémon are: Mew, Celebi, Jirachi, Deoxys, Phione, Manaphy,
  Darkrai, Shaymin, Arceus, Victini, Keldeo, Meloetta, Genesect,
  Diancie, Hoopa, Volcanion, Magearna, Marshadow, Zeraora.
2020-06-27 12:56:28 -07:00
rluzuriaga
f535b61341 Handle CLI load into existing tables gracefully (PR #300)
- Add exception handling for SQLite, PostgreSQL, and MySQL.
Sqlalchemy gives different exceptions for each database engine so I have
seperated each exception for each engine.
The error message passed by each engine is also different so that is
checked as well.
- Once the function gets one of those exceptions the following message
is outputed and the program ends: `ERROR:  Table 'TABLENAME' already
exists in the database. Did you mean to use 'pokedex load -D'`
- If the error is not that a table already exists, then the output will
just be the full error from sqlalchemy. That way if someone runs into
that error it can be reported as an issue and whoever can help would see
the full error.
---
Resolves part of issue #16
2020-06-17 16:12:32 -07:00
Andrew Ekstedt
ad429ab128 Fix Roserade's Central Kalos dex number and add tests
Roserade and Roselia were mistakenly sharing dex number 72. Presumably
an error from the original X/Y spreadsheet. I guess we never reripped
these after 3DS rom dumping became possible.

Regardless of how it happened, add a test and some database constraints
to ensure that it doesn't happen again.

- Add a test for gaps in pokedex numbers

- Add uniqueness constraints to pokemon_dex_numbers.  A species can
  only appear once per pokedex, and a number cannot be used more than
  once per pokedex.
2019-07-16 18:12:58 -07:00
Kip Yin
11f972f817 Catch StopIteration explicitly in group_by_object generator (#264)
Python 3.7 changed the behaviour of generators so that a StopIteration exception which bubbles up inside a generator is transformed into a RuntimeException. This means that when calling next() inside a generator, we have to explicitly catch the StopIteration and exit the function instead of relying on it to implicitly stop the generator. Otherwise the program will crash. Sigh.

For more information, see PEP 479.

https://www.python.org/dev/peps/pep-0479/
https://stackoverflow.com/questions/51700960/runtimeerror-generator-raised-stopiteration-every-time-i-try-to-run-app/51701040#51701040
2019-07-15 13:23:37 -07:00
Andrew Ekstedt
db824220d0
Merge pull request #238 from jrubinator/record-gift-pokemon
Record gift pokemon
2019-07-12 19:40:29 -07:00
Andrew Ekstedt
2fefb9176b Fix documentation for Language.iso639 and iso3166
ISO 639 is the language code and ISO 3166 is the country code, not the
other way around.
2019-07-01 21:33:06 -07:00
Andrew Ekstedt
59fd27c574 Require location identifiers to be unique.
Also require location area identifiers to be unique to their location.

I almost added a duplicate pokemon-league location when adding S/M
locations. This should help.
2018-09-29 10:53:18 -07:00
Jonathan Rubin
8694c8464a gift-pokemon: Document empty encounter_slots
I'm going to record gift-pokemon as having no encounter slot, which I
believe is correct by-and-large (since you always get the same mon).

There are a few randomized gift pokemon which do have some type
of encounter slot (eg. Crystal's Odd Egg). Those aren't tracked (yet).
2018-09-19 13:58:25 -04:00
Andrew Ekstedt
6d4d129c84 Add subtitle field to Location
Starting in Gen VI, locations have been able to have an optional
subtitle. In-game, the subtitle is displayed on a second line under the
main name.

In X/Y this is used to give some locations an alternate,
flavorful name; for example, Kalos Route 2 is also known as Avance
Trail. In S/M this is mainly used to divide some large locations like
Hau'oli City into separate sections: e.g., the Beachfront, the Shopping
District, and the Marina.

This commit just adds the column; there are no data changes.

I suppose we'll need to go back and re-rip X/Y location names now.
2018-09-15 20:01:08 -07:00
skylar
d17a772b4f Resolve some Python 3 encoding issues on Windows (#244)
When you open a file in Python 3, it defaults to using the system charset to encode the file, which is typically UTF-8 on linux systems (good) but Windows-1251 on Windows (bad).

We need to add explicit encoding=utf-8 arguments to open() calls when we open CSV files for reading or writing. To complicate matters, the csv module works only with byte strings in Python 2, and only with unicode strings in Python 3, so we can't just blindly use `io.open` everywhere.
2018-08-22 23:01:58 -07:00
Andrew Ekstedt
0c61702213 pass an explicit bind to table.create()
This isn't really necessary, since pokedex.db.connect sets
metadata.bind, but it's better to be explicit.
2018-03-30 15:07:10 -07:00
Andrew Ekstedt
417bd1afa2 fix pypy build
PyPy (>= 3.6) was consistently erroring out during `pokedex load`,
with the error

    OperationalError: (_sqlite3.OperationalError) cannot commit transaction - SQL statements in progress (Background on this error at: http://sqlalche.me/e/e3q8)

It turns out PyPy was not garbage collecting something, causing a
database cursor to be left open. See the giant comment block for
details.
2018-03-30 15:06:34 -07:00
Andrew Ekstedt
e1bbe78b72 fix test suite under SQLAlchemy 1.2.x
* SQLAlchemy 1.0 introduced "baked queries" - a way to construct Query
 objects so that they can be cached and reused.

 * SQLAlchemy 1.2 changed lazyloaded columns to use baked queries under the
 hood.

 * Our MultilangQuery class attempts to set _default_language_id right
 before the query is executed by overriding the __iter__ method.

 * Baked queries bypass the __iter__ method and call a lower-level
 method, _execute_and_instances, directly.

 * This caused problems where _default_language_id wouldn't get set
 correctly on lazyloaded columns.

 * To fix, make MultilangQuery override the _execute_and_instances
 method instead of __iter__.

 * This is really just a stopgap: the root cause is that query params
 are not preserved across lazyloads.

Tested with SQLAlchemy 0.9.7, 1.1.18, and 1.2.5.

Updates #236.
2018-03-30 12:01:21 -07:00
John T. Wodder II
c3f566b2c9 Addressed all of pyflakes3's complaints 2016-11-24 21:29:58 +00:00
Philippe Matray
7612277ee4 Add identifiers to item_fling_effects.csv 2016-10-23 18:56:43 +02:00
Andrew Ekstedt
ad73c9d43b Add priority and target columns to the move changelog
In preparation for adding Gen V→VI changelog entries.

No data changes.
2016-09-25 13:18:58 -07:00
Philippe Matray
d6d916e281 Add a description of the shape picks from Bulbapédia
Add the french translation for this table
http://bulbapedia.bulbagarden.net/wiki/List_of_Pok%C3%A9mon_by_body_style
2016-08-17 13:18:51 +02:00
Eevee (Lexy Munroe)
54ea67a804 Fix pokedex dump under Python 2 AND 3, at the same time even 2015-11-04 20:10:27 -08:00
Eevee (Lexy Munroe)
e0f4d3be7a Fix pokedex dump under Python 2 2015-11-04 20:03:08 -08:00
Eevee (Alex Munroe)
0ff24b4dc8 Fix the CLI in py3 2015-10-05 16:29:21 -07:00
Eevee (Alex Munroe)
b76b74e7a6 Compat with Python 3.3+ 2015-10-05 08:12:23 -07:00
Andrew Ekstedt
7d1b8cc4f8 Improve table docs w.r.t. ids and game indices.
Went though the table docs and noted which tables have ids that correspond
to game ids, and explained why we sometimes have ids in the 10000s.

Since docs attached to id columns aren't displayed in the web documentation,
these edits all go in the class docstring, and i took the liberty of deleting
the id column docs from every table i touched.

Also: added some words about how to get to the species from pokemon and
pokemon_form.

Also: note that item flags are not official (they are a holdover from
brownkun) and item pockets are sort of kind of official (they originated from
D/P i believe, but i'm not sure how we've been assigning them for newer items).

Pertains to #99.
2015-06-17 16:38:16 -07:00
Andrew Ekstedt
93988d966c load: Use COPY FROM STDIN on PostgreSQL.
COPY FROM FILE requires database superuser permissions,
because of the obvious security implications.

COPY FROM STDIN has no such restriction.

Also do some cleanup while we're here.
2015-05-30 22:48:20 -07:00
Andrew Ekstedt
76ad6391fc Document Type.target_efficacies and Type.damage_efficacies.
I can't keep them straight.
2015-05-24 23:12:21 -07:00
Andrew Ekstedt
02b1968bc5 "Fix" MultilangQuery.
SQLAlchemy 1.0 sets bindpararms to their default values when loading
lazy-loaded columns. This is in contrast to the 0.9 behaviour of
ignoring our incongruous bindparam alltogether.

So MultilangQuery is still broken, but now it breaks in the same way as before.

While we're here, set the correct param in one of the multilang tests. Not that
it matters.
2015-05-23 18:26:09 -07:00
Lynn "Zhorken" Vaughan
7ddecd527d Call characteristics "characteristics". #132 2014-07-28 19:01:33 -04:00
Petr Viktorin
df945eb601 Fix drain/recoil name in MoveMeta
The column was named recoil, but positive values meant
drain.
Rename the column, and introduce a hybrid property for
recoil.
2014-07-09 10:47:36 -07:00
Andrew Ekstedt
29824c73f4 Tweak Pokemon width/height docstrings 2014-07-06 12:17:11 -07:00
Andrew Ekstedt
5f54b3057d Fix PokemonForm.form_order docstring 2014-07-06 12:17:11 -07:00
Andrew Ekstedt
19d7335399 Remove some lies about encounter slots 2014-07-05 16:50:43 -07:00
Andrew Ekstedt
159eea093a Address some XXX comments 2014-07-05 16:50:24 -07:00
Andrew Ekstedt
cd04629404 PEP 257, yo
Put the closing quote of single-line docstrings on the same line, and
end with a period.

Just tables.py for now.
2014-07-05 16:48:58 -07:00
Andrew Ekstedt
6f5abb9540 Put column descriptions where they belong
The Column class accepts a 'doc' argument. Use it.

And while we're at it, make them all unicode strings.

Performed by the following sed script:

   s/info=dict(description=u\?\("[^"]*"\))/doc=u\1/
   s/info=dict(description=u\?\('[^']*'\))/doc=u\1/
   s/\(\s*\)info=dict(description=u\?\("[^"]*"\), /\1doc=u\2,\n\1info=dict(/
   s/\(\s*\)info=dict(description=u\?\('[^']*'\), /\1doc=u\2,\n\1info=dict(/

   /info=dict(description=u\?\('[^']*'\),$/ {
       s//doc=u\1,/
       n
       s/^\s*/&info=dict(/
   }
2014-07-05 16:46:37 -07:00
Andrew Ekstedt
195cdd2ea2 Fix typos in table docs
Also, remove a stray detail=True from an info dict.
2014-07-04 12:43:03 -07:00
Andrew Ekstedt
16c1d2c278 Fix natures.game_index 2014-07-03 17:25:26 -07:00
Andrew Ekstedt
341f6bbc83 Add internal IDs for types 2014-07-03 16:55:33 -07:00
Lynn "Zhorken" Vaughan
f4b73bcd5a Clarify that natures.game_index is 1-indexed.
Given that nature was determined by pid % 25 for a long time, I think
this is important to note.
2014-06-21 22:50:12 -04:00
Lynn "Zhorken" Vaughan
a71e4a2c13 Add game indices for natures.
I'm assuming (and I'm 99% sure I'm right) that the internal order for
natures has never changed.
2014-06-21 22:27:56 -04:00
Lynn "Zhorken" Vaughan
7286ba0254 Delete all the blank rows in pokemon_form_names.
All the tests still pass and everything.  We literally did not need them
after that innerjoin=True got commented out.
2014-06-11 22:22:32 -04:00
Lynn "Zhorken" Vaughan
63dac87a40 Dump translations for official languages by default.
I don't really like this but ehhhhhhh, the system for prose
translations seems to be to keep them in csv/translations/, and I can't
figure out how you're supposed to DO that, plus judging by the age of
the single file that's in there, that seems to be where translations go
to die.
2014-06-01 19:44:32 -04:00
Andrew Ekstedt
8d42445ae0 Fix typo in Machine.machine_number docstring. 2014-05-19 21:31:58 -07:00
Lynn "Zhorken" Vaughan
2626cf6304 Use null power instead of 0/1 power.
<Zhorken> I'd like to motion (again) that 0 and 1 power both be stored
    as null
<Zhorken> and — or * be displayed based on damage class
<Zhorken> which is what determines 0 vs 1 anyway, with the arbitrary
    who-fucking-cares exception of Me First
<eevee> the ayes have it
<Zhorken> awesome
<eevee> that's a good idea i don't remember its being motioned the
    first time
<Zhorken> I definitely remember arguing it when B/W released
2014-05-19 21:37:48 -04:00
Andrew Ekstedt
99184a6fde Knock Unicode(256) down to Unicode(79)
MySQL again. Indexed keys have a maximum length of 767 bytes and unicode
is stored as 3 bytes per character. Since 256*3 = 768, that pushes us
just past the limit. Could lower the length to 255 characters, but let's
go further - currently only names and identifers have a length limit,
and if a name or identifier doesn't fit in a terminal then i don't want
to see it.

Fixes issue #123.
2014-05-13 23:26:07 -07:00
Andrew Ekstedt
9e3bfd7971 Use Unicode(256) for names and identifiers.
MySQL requires that text columns that are indexed or used as keys be
given a maximum length.

256 bytes should be enough for anyone.
2014-03-13 00:29:49 -07:00
Lynn "Zhorken" Vaughan
7462c0fc1b Put short_effect before effect in ability_prose.
This way it matches /all the other prose tables/.  Relatedly, all the
abilities I recently expanded on had short_effect and effect swapped,
so I fixed that too.
2014-03-12 12:18:37 -04:00
Lynn "Zhorken" Vaughan
308f5f4ed9 Use UnicodeText for all text columns. 2014-03-11 20:15:16 -04:00
Lynn "Zhorken" Vaughan
5ddbee886f Update the docstring on rewrite_long_table_names. 2014-02-21 17:59:52 -05:00