Debanjum Singh Solanky 5673bd5b96 Keep original formatting in compiled text entry strings ...

- Explicity split entry string by space during split by max_tokens
- Prevent formatting of compiled entry from being lost
- The formatting itself contains useful information
  No point in dropping the formatting unnecessarily,
  even if (say) the currrent search models don't account for it (yet)

2023-03-30 14:02:46 +07:00

6.8 KiB

Raw Blame History

View Raw