Suggestion: InputRef should have a public API to visit the raw input, such as full_slice()

[cnglen]

Since chumsky supports imperative-style using custom, if we are not able to lookback in custom, maybe it's not fully impreative.

I found slice() but I'm not able to construct the input parameters for slice():

• Using the current cursor location, find the boundary of previous UTF8 char byte.
• Create range of previous character we want to lookback
• using InputRef::slice() to lookback.

background: In the parsing of org-mode file

Previous char is used to valid a pattern. See https://orgmode.org/worg/org-syntax.html

In a few cases, an instance of an element or object must be preceded or succeeded by a certain pattern, which is not itself part of the element or object. These patterns are specified using the PRE and POST tokens respectively, like so:

Methods tried:

1. any().then(parser), has some problem and not valid. For example:

• Parsing relies on the state of the previous character (PRE). Note that PRE is not part of the current object. PRE might be part of the previous object, so parsing with pre.then(object) is problematic and not feasible. For example, in [[https://a.b][foo]]^{2}, the "]" before "^" is part of the Link in AST.

2. the state of PRE(previous char) needs to be maintained and managed manually and very carefully

• this method works, but the code is very dirty. You have to decide whether to update PRE in every sub-parser implementation
• RollbackState have to be used to avoid state corrupted in backtrack.
• the performance is slow.

3. use imperative-style parser of custom, the problem is that in the implementation of chumsky, it seems that we can't lookback to previous character have been consumed.

I can't find any API of chumsky to get the previous char, but the logic is simple.

• Using the cursor, find the boundary of previous UTF8 char byte, which is in patter of 0zzz_zzzz / 110y_yyyy / 1110_xxxx / 1111_0www.
• Since the utf8 char has 4 bytes at most, we look 4 bytes earlier at most.

In my code, the final parser if a prev_valid_parser implement by custom(), but I change one line of code in input.rs of InputRef Struct to let the prev_valid_parser work in my local Chumsky crate.

/// Internal type representing an input as well as all the necessary context for parsing.
pub struct InputRef<'src, 'parse, I: Input<'src>, E: ParserExtra<'src, I>> {
    cursor: I::Cursor,
    // pub(crate) cache: &'parse mut I::Cache,
    pub cache: &'parse mut I::Cache, // <--------------------------------
    
    pub(crate) errors: &'parse mut Errors<I::Cursor, E::Error>,
    pub(crate) state: &'parse mut E::State,
    pub(crate) ctx: &'parse E::Context,
    #[cfg(feature = "memoization")]
    pub(crate) memos: &'parse mut HashMap<(usize, usize), Option<Located<I::Cursor, E::Error>>>,
}

pub trait PrevInput<'src>: Input<'src> {
    unsafe fn prev(cache: &mut Self::Cache, cursor: & Self::Cursor) -> Option<Self::Token>;
}

impl<'src> PrevInput<'src> for &'src str {
    #[inline(always)]
    unsafe fn prev(this: &mut Self::Cache, cursor: & Self::Cursor) -> Option<Self::Token> {
        let idx_byte_current = *cursor;
        let mut prev_char = None;
        for i in 1..5 {
            if idx_byte_current<i { // at the start of Self::Cache
                break;
            }
            let idx_byte = idx_byte_current-i;

            // from is_utf8_char_boundary() 
            // This is bit magic equivalent to: b < 128 || b >= 192
            if ((this.as_bytes()[idx_byte]) as i8) >= -0x40 {
                let c = this.get_unchecked(idx_byte..)
                    .chars()
                    .next()
                    .unwrap_unchecked();
                prev_char = Some(c);
                break;
            }
        }
        prev_char
    }
}

pub trait PrevInputRef<'src, I> {
    fn prev(&mut self) -> Option<I::Token> where I: PrevInput<'src>;
}

impl<'src, 'parse, I: Input<'src>, E: extra::ParserExtra<'src, I>> PrevInputRef<'src, I> for  chumsky::input::InputRef<'src, 'parse, I, E> {
    #[inline(always)]
    fn prev(&mut self) -> Option<I::Token>
    where
        I: PrevInput<'src>,
    {
        // E0716
        let binding = self.cursor();
        let a = binding.inner();        
        let token = unsafe { I::prev(self.cache, a)};
        token
    }
}

// valid prev char using `f`
pub(crate) fn prev_valid_parser<'a, C: 'a, F: Fn(Option<char>)->bool + Clone>(
    f: F
) -> impl Parser<'a, &'a str, (), MyExtra<'a, C>> + Clone {
    custom(move |inp| {
        let before = inp.cursor();
        let maybe_prev = inp.prev();
        if f(maybe_prev) {
            Ok(())
        } else {
            Err(Rich::custom(
                inp.span_since(&before),
                format!("invalid PRE: {maybe_prev:?}"),
            ))            
        }
    })
}

I know let cache be public in InputRef is not a good idea. Can we add one API to let the InputRef to lookback? By the way, I'm not able to create a Cursor to use InputRef::slice().

Mabey Cursor::new(cursor:usize), thus we can use the slice() API?

[zesterer]

If you know what your input type is already, it should be possible to make use of that. For example, with a &[T] input, you can just directly index. Let me know whether this makes sense.

[cnglen]

If you know what your input type is already, it should be possible to make use of that. For example, with a &[T] input, you can just directly index. Let me know whether this makes sense.

If there is some method let me visit the raw input, it's helpful for my situation. Does chumsky alreay support this?

How to acess the raw input in custom() or others ways?

• At this moment, It seems only the inp: &mut InputRef, which doesn't have any public API for raw_input.
• Or create a parser, fn my_parser(my_input: &[T]). This is strange.

[zesterer]

Using slice_since should work:

let last_token = inp.slice_since(&input.cursor()).last();

[cnglen]

Using slice_since should work:

let last_token = inp.slice_since(&input.cursor()).last();

1. The problem is how to create a previous version Cursor based current inp.cursor(). It seems that Cursor doesn't have such public API.

let prev_cursor = Cursor(inp.cursor() - 1); // somthing like this,

2. Where is input in input.cursor() from, or it's inp?

If I use the code below, last_token will always be None.

// valid prev char using `f`
pub(crate) fn prev_valid_parser<'a, C: 'a, F: Fn(Option<char>)->bool + Clone>(
    f: F
) -> impl Parser<'a, &'a str, (), MyExtra<'a, C>> + Clone {
    custom(move |inp| {
        
        let before = inp.cursor();

        // not work
        let last_token: &str = inp.slice_since(&before..); 
        let maybe_prev = last_token.chars().last(); // always return None!!!

        // work
        let maybe_prev_ok = inp.prev(); // using local public version self.cache in InputRef

        println!("maybe_prev: {:?} VS {:?}", maybe_prev, maybe_prev_ok);
        
        if f(maybe_prev) {
            Ok(())
        } else {
            Err(Rich::custom(
                inp.span_since(&before),
                format!("invalid PRE: {maybe_prev:?}"),
            ))            
        }
    })
}

[cnglen]

Suggestion: make full_slice() public API @zesterer

input.rs -> InputRef -> pub fn full_slice(&mut self)

Situation

Even if there is a public API to construct Cursor, we have to make sure the index is in the utf8 bounary for &str input, so we have to get raw input.

Another way is use &[char], but for my situation, lots of code has to be updated for data transformation from &[char] <-> &str, for me, this method is NOT a good idea.

So can we make the "full_slice()" from pub(crate) to pub, as below, so the user can visit raw_input in their own way.

Change and Example

Change to public: src/input.rs InputRef

    #[cfg_attr(not(feature = "regex"), allow(dead_code))]
    #[inline]
    pub fn full_slice(&mut self) -> I::Slice  <- See here ....................................................................
    // pub(crate) fn full_slice(&mut self) -> I::Slice        
    where
        I: SliceInput<'src>,
    {
        I::full_slice(self.cache)
    }

Example of user's code using full_slice():

pub(crate) fn prev_valid_parser<'a, C: 'a, F: Fn(Option<char>) -> bool + Clone>(
    f: F,
) -> impl Parser<'a, &'a str, (), MyExtra<'a, C>> + Clone {
    custom(move |inp| {
        let before = inp.cursor();
        let idx_byte_current = before.inner();
        let maybe_prev = get_prev_char_index(inp.full_slice(), *idx_byte_current); // user's function

        if f(maybe_prev) {
            Ok(())
        } else {
            Err(Rich::custom(
                inp.span_since(&before),
                format!("invalid PRE: {maybe_prev:?}"),
            ))
        }
    })
}

[zesterer]

Are you using an LLM to generate these responses?

[cnglen]

Are you using an LLM to generate these responses?

of course not.

[zesterer]

If you want to make a PR for this, I'd be happy to look it over.