Debanjum Singh Solanky fbb7747dcc Read Markdown file as utf8 instead of the default encoding used by OS ...

- Background
  1. Obsidian stores markdown notes as utf8[1]
  2. By default, the python `open' command uses the OS locale encoding[2]

  This was causing the `UnicodeDecodeError: <locale_encoding> codec can't decode byte' error

- Fix
  - Read markdown files as utf8
    The Obsidian plugin is the main use-case for markdown files in
    khoj currently and that stores md files as utf8.
    Do not assume utf8 for other content types like org-mode, beancount for now.
  - Fail if error in reading file as utf8, instead of ignoring errors.
    Would rather have user realize that their files are not going to
    get indexed correctly.

[1]: https://forum.obsidian.md/t/better-handle-md-files-not-stored-in-utf8-format/13524/3
[2]: https://docs.python.org/3/library/functions.html#open

2023-02-06 21:04:50 -03:00

..

conversation

Increase max_tokens returned by GPT summarizer. Remove default params

2023-01-16 22:55:36 -03:00

ledger

Use List, Tuple, Set from typing to support Python 3.8 for khoj

2023-02-06 01:23:52 -03:00

markdown

Read Markdown file as utf8 instead of the default encoding used by OS

2023-02-06 21:04:50 -03:00

org_mode

Use List, Tuple, Set from typing to support Python 3.8 for khoj

2023-02-06 01:23:52 -03:00

__init__.py

Move application files under src directory. Update Readmes

2021-08-17 04:11:03 -07:00

text_to_jsonl.py

Use List, Tuple, Set from typing to support Python 3.8 for khoj

2023-02-06 01:23:52 -03:00