Concept:
The given linear differential equation can be written using differential operator notation \( D = \frac{d}{dx} \):
\[
(D^2 - 4D)y = x^2 e^{2x}
\]
The particular integral \( y_p \) is found using the inverse operator:
\[
y_p = \frac{1}{D^2 - 4D} \left( x^2 e^{2x} \right)
\]
To evaluate this when an exponential function \( e^{ax} \) is multiplied by another function, we use the exponential shift rule:
\[
\frac{1}{f(D)} \left( e^{ax} V(x) \right) = e^{ax} \frac{1}{f(D+a)} V(x)
\]
Step 1: Setting up the particular integral operator expression.
Identify our function of the differential operator from the differential equation:
\[
f(D) = D^2 - 4D
\]
Here, our exponential multiplier term has a power coefficient of \( a = 2 \), and the remaining function is \( V(x) = x^2 \). Applying the shift rule changes the operator \( D \rightarrow D + 2 \):
\[
y_p = e^{2x} \frac{1}{(D+2)^2 - 4(D+2)} (x^2)
\]
Step 2: Simplifying the shifted denominator operator.
Let us expand the algebraic terms in the denominator:
\[
(D+2)^2 = D^2 + 4D + 4
\]
\[
-4(D+2) = -4D - 8
\]
Adding these expressions together:
\[
f(D+2) = (D^2 + 4D + 4) + (-4D - 8) = D^2 - 4
\]
Substitute this back into the expression for \( y_p \):
\[
y_p = e^{2x} \frac{1}{D^2 - 4} (x^2) \quad \cdots (1)
\]
Step 3: Expanding the operator via binomial expansion for polynomial terms.
To apply the operator to a polynomial like \( x^2 \), we rewrite the denominator in the form \( -(4 - D^2) \) and factor out the constant to use a binomial series:
\[
\frac{1}{D^2 - 4} = \frac{1}{-4 \left(1 - \frac{D^2}{4}\right)} = -\frac{1}{4} \left(1 - \frac{D^2}{4}\right)^{-1}
\]
Using the binomial expansion formula \( (1-t)^{-1} = 1 + t + t^2 + \cdots \):
\[
\left(1 - \frac{D^2}{4}\right)^{-1} = 1 + \frac{D^2}{4} + \frac{D^4}{16} + \cdots
\]
Since our target polynomial expression is \( x^2 \), any derivative higher than the second derivative will equal zero. Therefore, we can drop terms containing \( D^4 \) and higher powers:
\[
y_p = e^{2x} \left[ -\frac{1}{4} \left( 1 + \frac{D^2}{4} \right) \right] (x^2)
\]
Step 4: Distributing the operator onto the polynomial \( x^2 \).
Now, let's apply the operations inside the brackets to the polynomial:
\[
y_p = -\frac{1}{4} e^{2x} \left[ x^2 + \frac{1}{4} D^2(x^2) \right]
\]
Compute the required derivatives of \( x^2 \):
\[
D(x^2) = \frac{d}{dx}(x^2) = 2x
\]
\[
D^2(x^2) = \frac{d}{dx}(2x) = 2
\]
Substitute \( D^2(x^2) = 2 \) back into the expression:
\[
y_p = -\frac{1}{4} e^{2x} \left( x^2 + \frac{1}{4} \cdot 2 \right) = -\frac{1}{4} e^{2x} \left( x^2 + \frac{1}{2} \right)
\]
Step 5: Equating to find the target function \( y(x) \).
The problem states that the particular integral is written in the form \( y_p = e^{2x} y(x) \). Comparing this directly with our calculated result:
\[
e^{2x} y(x) = e^{2x} \left[ -\frac{1}{4} \left( x^2 + \frac{1}{2} \right) \right]
\]
Dividing out the matching exponential factor from both sides isolates our target function:
\[
y(x) = -\frac{1}{4} \left( x^2 + \frac{1}{2} \right)
\]
This matches option (A).