With the continuous development of conversational artificial intelligence, Emotion Recognition in Conversation (ERC) has garnered increasing attention. Psychological studies have shown that the speaker's self-emotional dependence and the influence between speakers are two core factors in sentiment analysis within conversations. Existing works typically generate responses to target utterances through training models, aiming to identify emotions by leveraging contextual information. However, incorporating additional context into target sentences complicates the task for language models to accurately identify the crucial information within the target sentences. To address this issue, we propose a novel model that separately models the speaker's self-emotion and the influence between speakers. Specifically, for the recognition of target utterances, we design prompt tailored to the task of Emotion Recognition in Conversation and integrate them with extracted keywords to form prompt sentences. Furthermore, to accurately capture the speaker's self-emotional state, we conduct analyses from both local and global perspectives, thereby capturing the speaker's emotional inertia. Extensive experiments conducted on four ERC datasets demonstrate the superiority of our proposed method.